Distributed resource management and scheduling method based on hybrid architecture

Abstract

The invention discloses a distributed resource management and scheduling method based on a hybrid architecture, and the method comprises the steps: starting a global scheduler and a local scheduler of each node, and enabling the nodes to automatically register the resource information of the nodes to a central registry; submitting a task by a user, and defining a resource demand, a priority and a constraint condition of the task; the global scheduler allocates tasks to appropriate nodes according to resource requirements and priorities; receiving a task by a local scheduler of the selected node, and generating initial scheduling according to a local resource condition and a constraint condition; the local scheduler allocates tasks to specific resources and executes the tasks, and the global scheduler, the local scheduler and the local scheduler dynamically adjust resource allocation and task scheduling according to real-time monitoring data and a time sequence prediction model; fault nodes are found in time through a heartbeat detection mechanism, and tasks on the fault nodes are redistributed to other available nodes. According to the method, efficient resource management and task scheduling are realized through a multi-stage scheduling architecture and multiple communication mechanisms.

Description

Technical Field

[0001] The present invention relates to the field of resource management and scheduling, and in particular to a distributed resource management and scheduling method based on a hybrid architecture. Background Art

[0002] Resource management and scheduling are key to ensuring efficient operation of the system. Existing distributed scheduling solutions are mainly pure distributed systems, characterized by direct communication between agents and no single decision maker. The scheduling problem is decomposed into multiple sub-problems, and local decision makers coordinate and solve them through communication mechanisms to improve the system's responsiveness and flexibility.

[0003] However, although the traditional centralized scheduling method has the advantage of a global view, it is prone to performance bottlenecks and single point failure problems in large-scale systems. Although the fully decentralized scheduling method can improve the response speed and flexibility of the system, it is difficult to achieve global optimization. Therefore, a new method that combines the advantages of centralized and decentralized methods is needed to solve these problems. Summary of the invention

[0004] In view of the above situation, the present invention provides a distributed resource management and scheduling method based on a hybrid architecture, which realizes efficient resource management and task scheduling through a multi-level scheduling architecture and multiple communication mechanisms.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] In one embodiment of the present invention, a distributed resource management and scheduling method based on a hybrid architecture is proposed, the method comprising:

[0007] Use service discovery tools to automatically discover nodes and services; automatically register resource information to the central registry when a node starts, and automatically deregister when it is shut down or fails;

[0008] Users submit tasks, defining their resource requirements, priorities, and constraints;

[0009] The global scheduler assigns tasks to appropriate nodes based on resource requirements and priorities;

[0010] The local scheduler of the selected node receives the task and generates an initial schedule based on local resource conditions and constraints; it optimizes the schedule by coordinating with the global scheduler and local scheduler;

[0011] The local scheduler assigns tasks to specific resources and executes them, while monitoring task execution and resource usage in real time;

[0012] Based on real-time monitoring data and time series prediction models, the global scheduler, local scheduler, and local scheduler dynamically adjust resource allocation and task scheduling;

[0013] The heartbeat detection mechanism can detect faulty nodes in a timely manner and reallocate tasks on the faulty nodes to other available nodes to ensure continuous execution of tasks.

[0014] Furthermore, a resource controller is used to perform fine-grained division of CPU and memory to ensure that each task obtains the required resources; resource information of each node is collected, the priority of tasks is planned according to the total amount of resources, the upper and lower limits of resources of each node and the current resource usage are identified to calculate the upper and lower limits of available resources of each node, and the usage quota of each node is allocated according to the resources required for the task and the upper and lower limits of available resources of each node.

[0015] Furthermore, the task status and related information are stored in the distributed storage system; when a node failure is detected, all task information on the node is obtained from the distributed storage system; according to the task priority and resource requirements, all tasks on the node are migrated to other available nodes; tasks are rescheduled using a global scheduler or a local scheduler to ensure that the new node has sufficient resources to execute these tasks; during the task migration process, ensure that the task status and progress can be correctly restored on the new node, and use distributed locks or other consistency mechanisms to ensure the consistency of the task status; record the information of the failed node and the task migration process, and use monitoring tools to monitor the status of nodes and tasks in real time.

[0016] Furthermore, nodes are automatically added or removed based on resource usage. If the current node resources are fully occupied, they are automatically removed. Based on historical data, time series prediction models, and user demand forecast load, resource expansion is performed in advance.

[0017] Furthermore, the global scheduler performs global resource allocation based on the resource conditions and task requirements of all nodes to ensure optimal allocation of tasks globally.

[0018] Furthermore, the global scheduler uses a centralized scheduling strategy to assign tasks to appropriate nodes based on resource requirements and priorities, and sends task information to the corresponding local scheduler; the local scheduler manages local resources on the node and performs fine-grained task scheduling based on the actual situation and constraints of the node; the local scheduler uses a decentralized scheduling strategy to optimize local task scheduling.

[0019] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned distributed resource management and scheduling method based on the hybrid architecture is implemented.

[0020] In one embodiment of the present invention, a computer-readable storage medium is further proposed, wherein the computer-readable storage medium stores a computer program for executing a distributed resource management and scheduling method based on a hybrid architecture.

[0021] Beneficial effects:

[0022] 1. The present invention ensures that resources are fully utilized and waste is avoided through fine-grained resource division and real-time monitoring.

[0023] 2. The present invention combines centralized and decentralized scheduling mechanisms to improve the response speed and flexibility of the system.

[0024] 3. The present invention ensures the stability and reliability of the system through fault detection and recovery mechanism.

[0025] 4. The present invention supports automatic scaling and predictive expansion, can adapt to systems of different scales and load changes, and is suitable for multiple application scenarios.

[0026] 5. The global scheduler of the present invention provides a global perspective to ensure global optimization of task scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a distributed resource management and scheduling method based on a hybrid architecture of the present invention;

[0028] Figure 2 It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION

[0029] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0030] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a device, apparatus, equipment, method or computer program product. Therefore, the present disclosure may be implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0031] According to an embodiment of the present invention, a distributed resource management and scheduling method based on a hybrid architecture is proposed, aiming to improve the responsiveness, resource utilization and flexibility of the system. The method realizes efficient resource management and task scheduling through a multi-level scheduling architecture and multiple communication mechanisms.

[0032] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0033] Figure 1 Schematic diagram of the distributed resource management and scheduling method based on hybrid architecture of the present invention. Figure 1 As shown, the method includes:

[0034] 1. Initialize the system

[0035] Start the global scheduler and the local scheduler of each node.

[0036] Use service discovery tools to automatically discover nodes and services.

[0037] A node automatically registers its resource information to the central registry when it starts, and automatically deregisters when it shuts down or fails.

[0038] 2. Task submission

[0039] Users submit tasks, defining their resource requirements, priorities, and constraints.

[0040] 3. Global Scheduling

[0041] The global scheduler assigns tasks to appropriate nodes based on resource requirements and priorities.

[0042] 4. Local Scheduling

[0043] The local scheduler of the selected node receives the task and generates an initial schedule based on the local resource conditions and constraints.

[0044] Optimize the scheduling plan by coordinating with the global scheduler and the local scheduler.

[0045] 5. Task Execution

[0046] The local scheduler assigns tasks to specific resources and executes them.

[0047] Monitor task execution and resource usage in real time.

[0048] 6. Dynamic Adjustment

[0049] Based on real-time monitoring data and time series prediction models, the global scheduler, local scheduler, and local scheduler dynamically adjust resource allocation and task scheduling.

[0050] 7. Troubleshooting

[0051] The heartbeat detection mechanism can be used to detect faulty nodes in a timely manner.

[0052] Redistribute tasks on failed nodes to other available nodes to ensure continuous execution of tasks.

[0053] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0054] In order to explain the above-mentioned distributed resource management and scheduling method based on the hybrid architecture more clearly, a specific embodiment is used for illustration below. However, it should be noted that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation to the present invention.

[0055] The process of implementing a distributed resource management and scheduling method based on a hybrid architecture is as follows:

[0056] 1. Dynamic resource discovery and registration---confirming available resources

[0057] Use service discovery tools (such as Consul, Etcd) to automatically discover nodes and services; automatically register resource information to the central registry when a node starts, and automatically log out when it is shut down or fails. Node: Computing resources such as physical servers, virtual machines, or containers, each of which has a certain computing power. Service: A group of applications or functional modules running on a node, such as a web server, database service, or message queue service.

[0058] Specifically:

[0059] (1) Configure the cluster mode of Consul or Etcd on each node (such as a physical server, virtual machine, or container) in the distributed system (a cluster consisting of multiple instances is used to avoid single point failure; even if a node fails, other nodes can still continue to provide services, and multiple instances in the cluster can back up each other to ensure the continuity of data and services) to ensure high availability and fault tolerance. Start Consul Agent or Etcd Agent on each node to join the existing service discovery cluster (similar to a database to determine the available number of computing resources); the service on each node (such as computing node, storage node, etc.) registers its information with Consul or Etcd through the API when it starts, where the registered information includes service identifier, IP address, port, health check URL, etc. Consul Agent or Etcd Agent is the client component of the service discovery and configuration management tool. They run on each node and are responsible for communicating with the central registry (i.e., Consul or Etcd cluster). When recognizing that the service on each node (such as Web service, database service, computing node, storage node, etc.) is started, it collects relevant information from the node registration.

[0060] (2) Other nodes or services can discover available services by querying Consul or Etcd. For example, use Consul's DNS interface or HTTP API to obtain a list of services and detailed information.

[0061] (3) The node startup script contains registration logic, calling the Consul or Etcd API to register the node information to the central registry. The registered information includes node ID, IP address, port, resource type (CPU, memory, storage, etc.), resource capacity, etc.

[0062] (4) The node shutdown script contains deregistration logic, which calls the Consul or Etcd API to remove the node information from the central registry. The heartbeat mechanism is used to detect the node status. If the node is unresponsive for a long time, the node is automatically deregistered.

[0063] Heartbeat mechanism: Each node regularly sends heartbeat signals to Consul or Etcd to indicate that it is still online. If a node does not send a heartbeat signal for a long time, Consul or Etcd will automatically mark it as unavailable and trigger the corresponding processing logic.

[0064] Consul or Etcd has a built-in automatic deregistration mechanism. When a node does not send a heartbeat for a long time, it will be automatically removed from the registry. You can configure a TTL (Time To Live) check, and if the node fails the health check within the specified time, it will be automatically deregistered.

[0065] 2. Fine-grained resource division---resource division according to tasks (each task contains corresponding memory space, system resources and life cycle, and the corresponding resource utilization is different)

[0066] (1) CPU and memory: Use cgroups (resource controller) to perform fine-grained division of CPU and memory to ensure that each task (a process, service, or application running in the system, any computing task that requires resources) obtains the required resources.

[0067] Cgroups are used to limit, record, and isolate resource usage of process groups (such as CPU, memory, disk I / O, etc.). Through cgroups, specific resource quotas can be assigned to each task to ensure that it does not occupy too many resources.

[0068] (2) Storage and network: Collect resource information of each node and plan the priority of tasks according to the total amount of resources. For example, prioritize tasks according to the size of resources required, give priority to tasks that use less resources, identify the upper and lower limits of resources for each node and the current resource usage, calculate the upper and lower limits of available resources for each node, and allocate the usage quota of each node according to the resources required by the task and the upper and lower limits of available resources for each node to avoid resource abuse.

[0069] 3. Resource monitoring and management

[0070] Monitor resource usage in real time and provide feedback to the central server of the distributed storage system for timely adjustments.

[0071] Fault detection and recovery: The heartbeat detection mechanism can detect faulty nodes in a timely manner and automatically migrate tasks to other available nodes.

[0072] The heartbeat detection mechanism is used to monitor the health status of nodes. If a node does not respond to the heartbeat signal for a long time, it is considered that the node has failed. Once a failed node is detected, the tasks on the node are automatically migrated to other available nodes to ensure the continuous execution of tasks and high availability of the system.

[0073] Configure heartbeat detection:

[0074] (1) Configure a heartbeat detection mechanism on each node to periodically send heartbeat signals to a central registry (such as Consul, Etcd, or Zookeeper).

[0075] (2) Set the heartbeat interval (for example, once every 5 seconds) and the timeout period (for example, if no heartbeat is received within 30 seconds, the node is considered faulty).

[0076] (3) Heartbeat signal sending:

[0077] Each node starts a background process or scheduled task to send heartbeat signals regularly.

[0078] The heartbeat signal can be a simple HTTP request or a custom protocol.

[0079] (4) The central registry periodically checks the heartbeat signal of the node. If no heartbeat is received within the timeout period, the node is marked as unavailable. A TTL (Time To Live) check can be configured. If the node fails the health check within the specified time, it will be automatically deregistered.

[0080] (5) When the central registry detects a node failure, the corresponding fault handling logic is triggered, and the scheduler or other related components are notified through an event notification mechanism (such as Consul's Watch API) or a message queue (such as Kafka, RabbitMQ).

[0081] Task information backup:

[0082] (1) Store the task status and related information in a distributed storage system (such as Redis, Cassandra) so that it can be quickly recovered when a node fails.

[0083] Task information should include task ID, current status, execution progress, etc.

[0084] (2) Processing after fault detection:

[0085] When a node failure is detected, all task information on the node is obtained from the distributed storage system.

[0086] Reassign these tasks to other available nodes based on their priority and resource requirements.

[0087] (3) Task Migration:

[0088] Reschedule the task using the global scheduler or the local scheduler.

[0089] Make sure the new node has enough resources to perform these tasks.

[0090] (4) Task status synchronization:

[0091] During the task migration process, ensure that the task status and progress can be correctly restored on the new node. Use distributed locks or other consistency mechanisms to ensure the consistency of task status.

[0092] (5) Record the information of failed nodes and the process of task migration to facilitate subsequent analysis and auditing. Use monitoring tools (such as Prometheus and Grafana) to monitor the status of nodes and tasks in real time.

[0093] 4. Elastic expansion

[0094] Automatic scaling: Automatically add or remove nodes based on resource usage. If the current node resources are fully occupied, they will be automatically removed to avoid wasting detection time by querying again.

[0095] Predictive expansion: Based on historical data, time series forecasting models, and user demand forecast load, resources can be expanded in advance.

[0096] (1) Data preprocessing:

[0097] Cleaning: Remove invalid or abnormal data.

[0098] Standardization: Standardize data of different dimensions so that they are on the same scale.

[0099] Feature engineering: extract useful features, such as the periodicity and trend of time series.

[0100] (2) The model uses a time series prediction model:

[0101] Training method:

[0102] The model is trained using the training set, the model performance is evaluated using the test set, and as new data accumulates, the model is retrained periodically to maintain its accuracy.

[0103] Time series forecasting model structure:

[0104] Input layer: receives time series data.

[0105] LSTM layer: contains multiple LSTM units, each of which consists of an input gate, a forget gate, an output gate, and a cell state.

[0106] Fully connected layer: Converts the output of the LSTM layer into the final prediction value.

[0107] LSTM unit structure:

[0108] Input gate: decides which information needs to be updated.

[0109] Forget gate: decides which information to discard.

[0110] Cell state: storing long-term memory.

[0111] Output gate: determines which information needs to be output.

[0112] 5. Hybrid Scheduling Architecture

[0113] Global scheduler: responsible for overall resource allocation and global optimization, using a centralized scheduling strategy.

[0114] Global resource allocation is performed based on the resource conditions and task requirements of all nodes in the system to ensure the optimal distribution of tasks globally. Considering factors such as load balancing and resource utilization, centralized algorithms and strategies are used to determine the initial distribution of tasks.

[0115] Specifically:

[0116] (1) Resource collection and monitoring:

[0117] Use service discovery tools (such as Consul, Etcd) to regularly collect resource information of each node.

[0118] Use monitoring tools (such as Prometheus and Grafana) to monitor node resource usage in real time.

[0119] (2) Task reception and preprocessing:

[0120] Receives tasks from users or applications.

[0121] Preprocess the tasks and extract information such as task resource requirements and priorities.

[0122] (3) Global Scheduling Algorithm:

[0123] Load balancing: Use load balancing algorithms (such as minimum number of connections, polling, weighted polling, etc.) to assign tasks to appropriate nodes.

[0124] The principle of the weighted round-robin algorithm is to assign a weight value to each node and assign tasks according to the proportion of the weight value. For example, if the weight of node A is 2 and the weight of node B is 3, then when assigning tasks, about 2 out of every 5 tasks will be assigned to A and 3 will be assigned to B.

[0125] Advantages: The weight can be flexibly adjusted according to factors such as node performance and resource conditions to achieve more accurate load balancing. Effective task allocation can be performed for nodes with different performance in a heterogeneous environment, giving full play to the advantages of high-performance nodes while preventing low-performance nodes from being overloaded.

[0126] Optimization algorithm: Use heuristic algorithms (such as greedy algorithms, genetic algorithms, simulated annealing, etc.) for global optimization.

[0127] Among them, the greedy algorithm: in each iteration, a local optimal option is selected according to the current state and the greedy strategy; after each selection, the state of the problem is updated. This may include updating the set of selected options, the number of remaining resources, etc.

[0128] Constraint satisfaction: Ensure that task assignments satisfy various constraints (such as priority, time windows, resource limits, etc.).

[0129] The time window includes the start time, end time, latest start time, latest end time, etc. of the task. The time dependency between tasks needs to be considered. For example, some tasks can only start after other tasks are completed. A time monitoring mechanism should be established to monitor the execution progress of tasks in real time to ensure that the tasks are completed within the specified time window. If a task is delayed, timely adjustment measures should be taken, such as reallocating resources and adjusting task priorities. For tasks with time dependencies, scheduling algorithms such as the critical path method (CPM) or the program evaluation and review technique (PERT) can be used to determine the execution order and timing of tasks.

[0130] (4) Task allocation:

[0131] Assign tasks to selected nodes and send task information to the corresponding local scheduler.

[0132] Monitoring and Adjustment:

[0133] The central server of the distributed storage system monitors the task execution and system resource usage.

[0134] Dynamically adjust task allocation according to actual conditions to ensure system stability and performance.

[0135] (5) Local scheduler: The local scheduler on each node is responsible for local resource management and task scheduling, using a decentralized scheduling strategy. If there are many nodes and the resources of a central server cannot be fully covered, a local scheduler can be used for decentralized processing. The local scheduler is only responsible for resource processing on each node.

[0136] The local scheduler can improve the performance and efficiency of the node by optimizing the use of resources. For example, for CPU-intensive tasks, you can parallelize the tasks to take full advantage of multi-core CPUs; for memory-intensive tasks, you can use memory optimization techniques such as memory compression and cache management. Regularly clean up and organize resources, such as deleting unnecessary temporary files and releasing unused memory, to keep the node's resources in good condition.

[0137] The local scheduler receives tasks from the global scheduler or other nodes and puts the tasks into the local task queue. The task queue can be sorted according to the priority, arrival time, etc. High-priority tasks can be given priority to ensure that critical tasks can be executed in time.

[0138] The local scheduler assigns tasks to appropriate execution units based on the nature and resource requirements of the tasks. For example, computationally intensive tasks can be assigned to CPU cores for execution, while graphics processing tasks can be assigned to GPUs for execution.

[0139] By using task parallelization technology, we can make full use of the multi-core processor or distributed computing power of the node to execute multiple tasks simultaneously and improve the efficiency of task execution.

[0140] Monitor the progress of task execution and promptly handle abnormal situations during task execution, such as task failure, insufficient resources, etc. If a task has problems, you can try to reallocate resources or re-execute the task.

[0141] The local scheduler can adopt a variety of task scheduling strategies, such as First Come First Served (FCFS), Shortest Job First (SJF), Highest Response Ratio First (HRRN), etc. According to the resource status of the node and the characteristics of the task, select the appropriate scheduling strategy.

[0142] For tasks with high real-time requirements, a real-time scheduling strategy can be used to ensure that the task is completed within the specified time. For example, the earliest deadline first (EDF) or lowest slack first (LLF) scheduling algorithm can be used.

[0143] Consider the dependencies between tasks. For tasks with dependencies, schedule them in the order of dependencies to ensure the correct execution order of tasks.

[0144] 6. Multi-level scheduling strategy

[0145] First-level scheduler: Global scheduler, which assigns tasks to appropriate nodes based on resource requirements and priorities.

[0146] The second-level scheduler: the local scheduler performs fine-grained task scheduling based on the actual situation and constraints on the node.

[0147] Function: Local resource management: manage local resources on the node, including CPU, memory, storage and network bandwidth. Task scheduling: perform fine-grained task scheduling based on the actual situation and constraints of the node. Distributed scheduling strategy: use distributed algorithms and strategies to optimize local task scheduling.

[0148] Implementation steps:

[0149] (1) Resource management:

[0150] Use cgroups to manage CPU and memory resources.

[0151] Manage storage and network bandwidth using a quota system.

[0152] (2) Task acceptance and verification:

[0153] Receives tasks from the global scheduler.

[0154] Verify that the resource requirements of the task are consistent with the local resource situation.

[0155] (3) Local Scheduling Algorithm:

[0156] Priority scheduling: Scheduling based on task priority.

[0157] Preemptive scheduling: High-priority tasks can preempt the resources of low-priority tasks.

[0158] Fair scheduling: ensure that all tasks receive reasonable resource allocation.

[0159] (4) Task Execution:

[0160] Assign tasks to specific computing resources (such as containers and processes) and start the tasks.

[0161] Monitor the execution of tasks and record the status and logs of tasks.

[0162] (5) Resource recovery and redistribution:

[0163] When the task is completed, the resources it occupies are reclaimed.

[0164] Dynamically adjust resource allocation to respond to new task requirements.

[0165] 7. Communication Mechanism

[0166] Agents on the same level communicate or exchange information with each other to perform better scheduling for subproblems and select suboptimal strategies through cooperation mechanisms to achieve common goals.

[0167] The solution is gradually optimized by exchanging scheduling information between different types of agents to eliminate conflicts or modify existing schedules.

[0168] 8. Scheduling Algorithm

[0169] Constructive heuristics: Generate initial schedules using methods such as greedy algorithms, Earliest Due Date (EDD), etc.

[0170] Iterative heuristics: Continuously optimize the scheduling plan through methods such as taboo search and simulated annealing until the termination conditions are met.

[0171] Based on the above invention concept, Figure 2 As shown, the present invention also proposes a computer device 100, including a memory 110, a processor 120, and a computer program 130 stored in the memory 110 and executable on the processor 120. When the processor 120 executes the computer program 130, the aforementioned distributed resource management and scheduling method based on the hybrid architecture is implemented.

[0172] Based on the aforementioned inventive concept, the present invention further proposes a computer-readable storage medium, which stores a computer program for executing the aforementioned distributed resource management and scheduling method based on a hybrid architecture.

[0173] The distributed resource management and scheduling method based on the hybrid architecture proposed in the present invention has the following highlights:

[0174] (1) Efficient resource utilization: Through fine-grained resource division and real-time monitoring, resources are fully utilized to avoid waste.

[0175] (2) High responsiveness: The combination of centralized and decentralized scheduling mechanisms improves the system's response speed and flexibility.

[0176] (3) High reliability: Ensure system stability and reliability through fault detection and recovery mechanism.

[0177] (4) Scalability: It supports automatic scaling and predictive expansion, can adapt to systems of different scales and load changes, and is suitable for multiple application scenarios.

[0178] (5) Global optimization: The global scheduler provides a global perspective to ensure global optimization of task scheduling.

[0179] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements contained in the spirit and scope of the attached claims.

[0180] Regarding the limitation of the protection scope of the present invention, those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A distributed resource management and scheduling method based on a hybrid architecture, characterized in that: The method includes: Use service discovery tools to automatically discover nodes and services; automatically register resource information to the central registry when a node starts, and automatically deregister when it is shut down or fails; Users submit tasks, defining their resource requirements, priorities, and constraints; The global scheduler assigns tasks to appropriate nodes based on resource requirements and priorities; The local scheduler of the selected node receives the task and generates an initial schedule based on local resource conditions and constraints; it optimizes the scheduling plan by coordinating with the global scheduler and local scheduler; The local scheduler assigns tasks to specific resources and executes them, while monitoring task execution and resource usage in real time; Based on real-time monitoring data and time series prediction models, the global scheduler, local scheduler, and local scheduler dynamically adjust resource allocation and task scheduling; The heartbeat detection mechanism can detect faulty nodes in a timely manner and reallocate tasks on the faulty nodes to other available nodes to ensure continuous execution of tasks.

2. The distributed resource management and scheduling method based on hybrid architecture according to claim 1, characterized in that: Use resource controllers to perform fine-grained division of CPU and memory to ensure that each task obtains the required resources; collect resource information for each node, plan the priority of tasks based on the total amount of resources, identify the upper and lower limits of resources for each node and the current resource usage to calculate the upper and lower limits of available resources for each node, and allocate the usage quota for each node based on the resources required for the task and the upper and lower limits of available resources for each node.

3. The distributed resource management and scheduling method based on hybrid architecture according to claim 1, characterized in that: The task status and related information are stored in the distributed storage system. When a node failure is detected, all task information on the node is obtained from the distributed storage system. All tasks on the node are migrated to other available nodes according to the task priority and resource requirements. Use the global scheduler or local scheduler to reschedule tasks and ensure that the new node has sufficient resources to execute these tasks. During the task migration process, ensure that the status and progress of the task can be correctly restored on the new node, and use distributed locks or other consistency mechanisms to ensure the consistency of the task status. Record the information of failed nodes and the process of task migration, and use monitoring tools to monitor the status of nodes and tasks in real time.

4. The distributed resource management and scheduling method based on hybrid architecture according to claim 1, characterized in that: Automatically add or remove nodes based on resource usage. If the current node resources are fully occupied, they will be automatically removed. Based on historical data, time series prediction models, and user demand forecast load, resource expansion is performed in advance.

5. The distributed resource management and scheduling method based on hybrid architecture according to claim 1, characterized in that: The global scheduler performs global resource allocation according to the resource conditions and task requirements of all nodes to ensure the optimal allocation of tasks on a global scale.

6. The distributed resource management and scheduling method based on hybrid architecture according to claim 1, characterized in that: The global scheduler uses a centralized scheduling strategy to assign tasks to appropriate nodes based on resource requirements and priorities, and sends task information to the corresponding local scheduler; the local scheduler manages local resources on the node and performs fine-grained task scheduling based on the actual situation and constraints of the node; the local scheduler uses a decentralized scheduling strategy to optimize local task scheduling.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 6.

Images