Intelligent task scheduling method and device, equipment and storage medium
Through the registration center and the LASSO regression prediction algorithm dynamically allocates tasks, combined with the load balancing algorithm, the problems of uneven resource utilization and task delay in the existing task scheduling system are solved, and system performance and resource utilization are improved.
Patent Information
- Application Number
- CN202411202479.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-06
AI Technical Summary
There is an imbalance in resource utilization and task priority management in the existing task scheduling system, resulting in system performance degradation and resource waste, and the inability to dynamically adjust task priority and scheduling strategies. There is a lack of accurate prediction of task execution time, resulting in task delay and response. Time increases.
By registering the work nodes in the registration center, obtaining node information, using the decision center and LASSO regression prediction algorithm to predict the time-consuming execution of tasks on each node, dynamically allocating tasks to the node with the shortest execution time, and monitoring and adjusting task allocation in real time. Determine the node status by combining the load balancing algorithm.
It significantly improves the system's resource utilization, performance, scheduling accuracy and adaptability, and ensures the efficient and reliableness of large-scale distributed systems and complex task environments.
Smart Images

Figure CN119088522A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of resource management technology, and in particular to an intelligent task scheduling method, device, equipment and storage medium. Background Art
[0002] Existing task scheduling systems face multiple challenges, especially in terms of resource utilization and task priority management. Traditional task scheduling methods are usually based on static rules or simple priority algorithms, which often perform poorly when faced with dynamically changing system environments and complex task requirements.
[0003] In traditional task scheduling systems, tasks are usually assigned to various working nodes according to fixed rules, which may cause some nodes to be overloaded while other nodes are idle. Such uneven resource utilization will lead to system performance degradation and resource waste.
[0004] Many scheduling systems lack flexibility in handling tasks of different priorities and are unable to dynamically adjust the priority and scheduling strategies of tasks according to actual conditions. This may result in urgent tasks not being processed in a timely manner, affecting the overall performance of the system and user experience.
[0005] Traditional scheduling algorithms lack the ability to accurately predict task execution time and cannot accurately schedule tasks based on the actual needs of the tasks and the current status of the nodes, which can lead to task delays and increased system response time. Summary of the invention
[0006] The main purpose of the present invention is to provide an intelligent task scheduling method, device, equipment and storage medium, aiming to solve the technical problem of uneven resource utilization in existing task scheduling systems.
[0007] To achieve the above object, the present invention provides an intelligent task scheduling method, which comprises the following steps:
[0008] Register the working node to the registration center and obtain the working information of the working node in the registration center;
[0009] Receiving a task scheduling request for a task to be executed, and sending the task scheduling request to a decision center;
[0010] The decision center determines the execution time of the working node according to the task scheduling request and the working information of the working node, and allocates a target working node to the task to be executed according to the execution time of the working node.
[0011] Optionally, the step of determining the execution time of the working node according to the task scheduling request and the working information of the working node by the decision center, and allocating a target working node to the task to be executed according to the execution time of the working node includes:
[0012] Identify the task information of the task scheduling request, and use the LASSO regression prediction algorithm through the decision center to predict the execution time of the task to be executed on each working node according to the task information and the working information of the working node;
[0013] The working node with the shortest execution time is selected as the target working node of the task to be executed.
[0014] Optionally, after the step of identifying the task information of the task scheduling request, the method further includes:
[0015] The task information and the work information are cleaned and converted to remove data irrelevant to the task to be performed, and the task information and the work information are standardized or normalized to meet the input requirements of the LASSO regression prediction algorithm.
[0016] Optionally, the step of registering the working node with the registration center includes:
[0017] Determine whether the working node is available;
[0018] If so, the available working nodes are registered with a registration center, and general information of the working nodes is recorded by the registration center, where the general information includes the node type of the working node.
[0019] Optionally, the step of allocating a target working node for the task to be executed according to the execution time of the working node includes:
[0020] Determine a candidate working node that can execute the task according to the general information of the working node;
[0021] The candidate working node with the shortest execution time is used as the target working node of the task to be executed.
[0022] Optionally, after the step of allocating a target working node for the task to be executed according to the execution time of the working node, the method further includes:
[0023] Monitor the task execution status of the target working node and dynamically adjust task allocation according to task requirements.
[0024] Optionally, the step of allocating a target working node for the task to be executed according to the execution time of the working node includes:
[0025] The load status of the working node is determined by using a load balancing algorithm, and a target working node is allocated to the task to be executed according to the load status and execution time of the working node.
[0026] Furthermore, in order to achieve the above object, the present invention also provides an intelligent task scheduling device, the intelligent task scheduling device comprising:
[0027] A node registration module registers the working node to the registration center and obtains the working information of the working node in the registration center;
[0028] A task receiving module receives a task scheduling request for a task to be executed and sends the task scheduling request to a decision center;
[0029] The node allocation module determines the execution time of the working node according to the task scheduling request and the working information of the working node through the decision center, and allocates a target working node to the task to be executed according to the execution time of the working node.
[0030] Furthermore, to achieve the above-mentioned purpose, the present invention also provides an intelligent task scheduling device, which includes a memory, a processor, and an intelligent task scheduling program stored in the memory and executable on the processor, and the intelligent task scheduling program, when executed by the processor, implements the steps of the intelligent task scheduling method described above.
[0031] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a storage medium, on which an intelligent task scheduling program is stored, and when the intelligent task scheduling program is executed by a processor, the steps of the intelligent task scheduling method described above are implemented.
[0032] The present invention relates to an intelligent task scheduling method, which registers a working node to a registration center, obtains the working information of the working node in the registration center, receives a task scheduling request of a task to be executed, sends the task scheduling request to a decision center, determines the execution time of the working node according to the task scheduling request and the working information of the working node by the decision center, and allocates a target working node to the task to be executed according to the execution time. The present invention predicts the task execution time through real-time monitoring and data collection of working nodes, thereby selecting the optimal node for task scheduling, significantly improving the resource utilization, performance, scheduling accuracy, system robustness and adaptability of the system, and providing an efficient and reliable solution for large-scale distributed systems and complex task environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 A schematic diagram of the structure of the device hardware operating environment involved in the embodiment of the intelligent task scheduling device of the present invention;
[0034] Figure 2 This is a flow chart of the first embodiment of the intelligent task scheduling method of the present invention;
[0035] Figure 3 Schematic diagram of functional modules of a preferred embodiment of the intelligent task scheduling device of the present invention.
[0036] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0037] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0038] It should be noted that existing task scheduling systems face multiple challenges, especially in terms of resource utilization and task priority management. Traditional task scheduling methods are usually based on static rules or simple priority algorithms, which often perform poorly when faced with dynamically changing system environments and complex task requirements.
[0039] In traditional task scheduling systems, tasks are usually assigned to various working nodes according to fixed rules, which may cause some nodes to be overloaded while other nodes are idle. Such uneven resource utilization will lead to system performance degradation and resource waste.
[0040] Many scheduling systems lack flexibility in handling tasks of different priorities and are unable to dynamically adjust the priority and scheduling strategies of tasks according to actual conditions. This may result in urgent tasks not being processed in a timely manner, affecting the overall performance of the system and user experience.
[0041] Traditional scheduling algorithms lack the ability to accurately predict task execution time and cannot accurately schedule tasks based on the actual needs of the tasks and the current status of the nodes, which can lead to task delays and increased system response time.
[0042] Based on the above-mentioned defects, the present invention provides an intelligent task scheduling device, referring to Figure 1 , Figure 1 It is a structural diagram of the device hardware operating environment involved in the embodiment of the intelligent task scheduling device of the present invention.
[0043] like Figure 1As shown, the intelligent task scheduling device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0044] Those skilled in the art will understand that Figure 1 The hardware structure of the intelligent task scheduling device shown in the figure does not constitute a limitation on the intelligent task scheduling device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0045] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module and an intelligent task scheduling program. The operating system is a program for managing and controlling the intelligent task scheduling device and software resources, and supports the operation of the network communication module, the user interface module, the intelligent task scheduling program and other programs or software; the network communication module is used to manage and control the network interface 1004; the user interface module is used to manage and control the user interface 1003.
[0046] exist Figure 1 In the hardware structure of the intelligent task scheduling device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; the processor 1001 can call the intelligent task scheduling program stored in the memory 1005 and perform the following operations:
[0047] Register the working node to the registration center and obtain the working information of the working node in the registration center;
[0048] Receiving a task scheduling request for a task to be executed, and sending the task scheduling request to a decision center;
[0049] The decision center determines the execution time of the working node according to the task scheduling request and the working information of the working node, and allocates a target working node to the task to be executed according to the execution time of the working node.
[0050] Furthermore, the step of determining the execution time of the working node according to the task scheduling request and the working information of the working node by the decision center, and allocating a target working node for the task to be executed according to the execution time of the working node includes:
[0051] Identify the task information of the task scheduling request, and use the LASSO regression prediction algorithm through the decision center to predict the execution time of the task to be executed on each working node according to the task information and the working information of the working node;
[0052] The working node with the shortest execution time is selected as the target working node of the task to be executed.
[0053] Furthermore, after the step of identifying the task information of the task scheduling request, the method further includes:
[0054] The task information and the work information are cleaned and converted to remove data irrelevant to the task to be performed, and the task information and the work information are standardized or normalized to meet the input requirements of the LASSO regression prediction algorithm.
[0055] Furthermore, the step of registering the working node to the registration center includes:
[0056] Determine whether the working node is available;
[0057] If so, the available working nodes are registered with a registration center, and general information of the working nodes is recorded by the registration center, where the general information includes the node type of the working node.
[0058] Furthermore, the step of allocating a target working node for the task to be executed according to the execution time of the working node includes:
[0059] Determine a candidate working node that can execute the task according to the general information of the working node;
[0060] The candidate working node with the shortest execution time is used as the target working node of the task to be executed.
[0061] Furthermore, after the step of allocating a target working node for the task to be executed according to the execution time of the working node, the method further includes:
[0062] Monitor the task execution status of the target working node and dynamically adjust task allocation according to task requirements.
[0063] Furthermore, the step of allocating a target working node for the task to be executed according to the execution time of the working node includes:
[0064] The load status of the working node is determined by using a load balancing algorithm, and a target working node is allocated to the task to be executed according to the load status and execution time of the working node.
[0065] The specific implementation of the intelligent task scheduling device of the present invention is basically the same as the embodiments of the following intelligent task scheduling method, and will not be repeated here.
[0066] The present invention also provides an intelligent task scheduling method based on the intelligent task scheduling device.
[0067] Reference Figure 2 , Figure 2 Schematic diagram of the first embodiment of the intelligent task scheduling method of the present invention.
[0068] The embodiment of the present invention provides an embodiment of the intelligent task scheduling method. It should be noted that although the logical order is shown in the flow chart, in some cases, the steps shown or described may be executed in a different order than here.
[0069] In various embodiments of the intelligent task scheduling method, the execution subject is an intelligent task scheduling device.
[0070] The intelligent task scheduling method is applied to an intelligent task scheduling device, and the intelligent task scheduling method includes:
[0071] Step S10, registering the working node to the registration center, and obtaining the working information of the working node in the registration center;
[0072] Existing task scheduling systems face multiple challenges, especially in terms of resource utilization and task priority management. Traditional task scheduling methods are usually based on static rules or simple priority algorithms, which often perform poorly when faced with dynamically changing system environments and complex task requirements.
[0073] In traditional task scheduling systems, tasks are usually assigned to various working nodes according to fixed rules, which may cause some nodes to be overloaded while other nodes are idle. Such uneven resource utilization will lead to system performance degradation and resource waste.
[0074] Many scheduling systems lack flexibility in handling tasks of different priorities and are unable to dynamically adjust the priority and scheduling strategies of tasks according to actual conditions. This may result in urgent tasks not being processed in a timely manner, affecting the overall performance of the system and user experience.
[0075] Traditional scheduling algorithms lack the ability to accurately predict task execution time and cannot accurately schedule tasks based on the actual needs of the tasks and the current status of the nodes, which can lead to task delays and increased system response time.
[0076] In this embodiment, the working node is registered with the registration center, and the working information of the working node in the registration center is obtained. The specific steps may include:
[0077] Node startup: Each working node will perform self-checking and initialization when it starts to ensure that the basic functions and status of the node are normal. After the node is started, a data packet containing basic information such as node type (such as application node or database node), IP address, HOST name, etc. will be generated.
[0078] Information registration: The node sends a data packet containing its own information to the registration center to complete the node registration. For example, the node sends a data packet to the designated interface of the registration center through the REST API or message queue. After receiving the data packet, the registration center parses and stores the node information.
[0079] Information storage: The registration center stores the received node information in the database for subsequent query and use. For example, the registration center can use a relational database (such as MySQL) or a NoSQL database (such as MongoDB) to store node information to ensure data persistence and fast query.
[0080] Information acquisition: When the system needs to obtain information about working nodes, it can query the registration center to obtain the latest node status and configuration information. For example, the system can obtain node information through API or directly query the database, and process and use it as needed.
[0081] In addition, the step of obtaining the working information of the working node in the registration center may include:
[0082] The metadata collection component establishes a communication channel with each working node to ensure that the working information of the node can be obtained in real time or regularly. The metadata collection component is a system module (such as the Prometheus tool) that is used to collect and monitor the status and performance data of each working node to provide support for task scheduling and resource management. It obtains detailed working information of the node in real time or regularly, such as resource usage, task queue information, node status, etc., to ensure that the system can make decisions based on the latest data.
[0083] Among them, work information may include application service information, resource usage, database service information, node status, historical performance data, configuration parameters, etc.
[0084] Application service information: start time, duration, predicted duration of the currently executed task, waiting tasks in the queue, and predicted duration.
[0085] Resource usage: node memory usage, CPU usage, network bandwidth usage, and disk IO usage.
[0086] Database service information: memory usage, CPU usage, disk IO usage of the database service, statistics and index information of the data table.
[0087] Node status: whether the node is online and available, the node's operating status, and the overall load situation.
[0088] Historical performance data: historical task execution time, resource usage history.
[0089] Configuration parameters: hardware configuration, software version, network configuration.
[0090] Example description: The metadata collection component establishes a communication channel with the working node B through the HTTP protocol to obtain node status and task information in real time. Through the established HTTP channel, the metadata collection component periodically (for example, every 5 minutes) obtains the following information from node B:
[0091] Application service information: The start time of task T1 currently executed by node B is 10:00, the current duration is 5 minutes, and the predicted total duration is 15 minutes; there are 2 waiting tasks in the queue, and the predicted total durations are 10 minutes and 12 minutes respectively.
[0092] Resource usage: Node B's memory usage is 70%, CPU usage is 65%, and network bandwidth usage is 40%.
[0093] Database service information: The database service memory usage of node B is 60%, the CPU usage is 55%, and the disk IO usage is 50%; data table X has 1 million records, and the hit rate of index Y is 95%.
[0094] After the collected information of node B is packaged, it is transmitted to the central database through the network for storage, ensuring that the system can query and use the latest node working information at any time.
[0095] Step S20, receiving a task scheduling request for a task to be executed, and sending the task scheduling request to a decision center;
[0096] In this embodiment, the system receives a task scheduling request for a task to be executed, which is usually submitted by a user or automatically generated. The task scheduling request can be received in a variety of ways, such as through a user interface (UI), an API interface, an automated script, etc. The task scheduling request contains detailed information about the task, such as the task type, priority, resource requirements, start time, and estimated completion time.
[0097] The received task scheduling request is sent to the decision center for task allocation decision making. The system transmits the task scheduling request to the decision center through internal communication mechanisms (such as HTTP requests, message queues, RPC calls, etc.). After receiving the request, the decision center parses the task information and allocates tasks according to the current resource status and scheduling strategy of the system.
[0098] Example description: The system monitors scheduled tasks that need to be executed, such as daily scheduled database backup. At the scheduled time, the system automatically generates a task scheduling request, including the task type (such as database backup), priority (such as medium), resource requirements (such as 1 CPU and 2GB of memory), start time, and estimated completion time. The system's built-in scheduling module automatically generates task scheduling requests according to the scheduled schedule and saves the task information to the task queue.
[0099] The system extracts the task scheduling request to be executed from the task queue and sends the task information to the task allocation interface of the decision center through the message queue. The system packages the task type, priority, resource requirements, start time and estimated completion time, and transmits them to the decision center through the message queue. After receiving the message, the decision center parses the task information and makes a task allocation decision.
[0100] Step S30: The decision center determines the execution time of the working node according to the task scheduling request and the working information of the working node, and allocates a target working node to the task to be executed according to the execution time of the working node.
[0101] In this embodiment, the system receives a task scheduling request for a task to be executed, which contains detailed information of the task, such as task type, priority, resource requirements, etc. The system obtains the latest work information of all working nodes, which includes the CPU usage rate, memory usage rate, currently executing task list, task queue length, etc. of the node.
[0102] The decision center uses the LASSO regression prediction algorithm to predict the execution time of the task on each node based on the task information in the task scheduling request and the work information of the work node. The specific steps may include:
[0103] Feature selection: Select features that affect task execution time, such as CPU usage, memory usage, task queue length, etc.
[0104] Model training: Use historical task data to train the LASSO regression model and determine the weight of each feature.
[0105] Time prediction: Use the trained model to predict the execution time of new tasks on different nodes.
[0106] According to the predicted execution time, the work node with the shortest execution time can be selected as the target work node. The task allocation information is sent to the target node through the internal communication mechanism, and the task execution status is monitored in real time to ensure the smooth completion of the task.
[0107] Example description:
[0108] The user submits a data processing task through the task management interface. The task contains the following information:
[0109] Task type: Data processing
[0110] Priority: High
[0111] Resource requirements: 2 CPUs, 4GB of memory.
[0112] The system obtains the latest work information of all current working nodes from the central database:
[0113] Node A: CPU usage 70%, memory usage 65%, task queue length 3
[0114] Node B: CPU usage 50%, memory usage 55%, task queue length 2
[0115] Node C: CPU usage 80%, memory usage 75%, task queue length 4.
[0116] The decision center uses the LASSO regression prediction algorithm to predict the execution time of tasks on each node based on task information and node work information:
[0117] Node A: Predicted execution time is 15 minutes
[0118] Node B: predicted execution time is 10 minutes
[0119] Node C: The predicted execution time is 20 minutes.
[0120] Based on the predicted execution time, node B with the shortest execution time is selected as the target working node. The system assigns the task to node B and starts the execution of the task. The task start time is the current time and the estimated completion time is 10 minutes later.
[0121] The present invention relates to an intelligent task scheduling method, which registers a working node to a registration center, obtains the working information of the working node in the registration center, receives a task scheduling request of a task to be executed, sends the task scheduling request to a decision center, determines the execution time of the working node according to the task scheduling request and the working information of the working node by the decision center, and allocates a target working node to the task to be executed according to the execution time. The present invention predicts the task execution time through real-time monitoring and data collection of working nodes, thereby selecting the optimal node for task scheduling, significantly improving the resource utilization, performance, scheduling accuracy, system robustness and adaptability of the system, and providing an efficient and reliable solution for large-scale distributed systems and complex task environments.
[0122] Furthermore, a second embodiment of the intelligent task scheduling method of the present invention is proposed, wherein the above step S30 includes:
[0123] Step a1, identifying the task information of the task scheduling request, and using the LASSO regression prediction algorithm by the decision center to predict the execution time of the task to be executed on each working node according to the task information and the working information of the working node;
[0124] Step a2: Select the working node with the shortest execution time as the target working node of the task to be executed.
[0125] In this embodiment, when a task scheduling request is received, the detailed information of the task is first identified, including the task type, priority, resource requirements, etc. The task scheduling request is parsed, the task information therein is extracted, and it is passed to the decision center. The latest work information of all working nodes is obtained from the metadata collection component, which includes the CPU usage, memory usage, task queue length, etc. of the node.
[0126] The decision center uses the LASSO regression prediction algorithm to predict the execution time of the task on each node based on the task information in the task scheduling request and the work information of the work node.
[0127] LASSO (Least Absolute Shrinkage and Selection Operator) regression is a linear regression method that implements feature selection and model training by introducing the L1 regularization term. The objective function of LASSO regression is as follows:
[0128]
[0129] Among them, y i is the actual task execution time of the i-th sample; β0 is the intercept term; β j is feature x j The regression coefficient of x ijis the jth eigenvalue of the ith sample; n is the number of samples; p is the number of features; λ is the regularization parameter used to control the complexity of the model and the strength of feature selection.
[0130] Select features that affect task execution time, such as CPU usage, memory usage, task queue length, etc. Use historical task data to train the LASSO regression model and determine the weight of each feature.
[0131] Use the trained model to predict the execution time of new tasks on different nodes. Based on the predicted execution time, select the working node with the shortest execution time as the target working node.
[0132] Example description:
[0133] Assume there is an image processing task, which contains the following information:
[0134] Task type: Image processing;
[0135] Priority: Medium;
[0136] Resource requirements: 4 CPUs, 8GB of memory;
[0137] Get the work information of the work node.
[0138] The system obtains the latest work information of all current working nodes from the metadata collection component:
[0139] Node D: CPU usage 60%, memory usage 70%, task queue length 5;
[0140] Node E: CPU usage 55%, memory usage 50%, task queue length 3;
[0141] Node F: CPU usage 75%, memory usage 80%, task queue length 6.
[0142] The decision center uses the LASSO regression prediction algorithm to predict the execution time of tasks on each node based on task information and node work information:
[0143] Node D: predicted execution time is 25 minutes;
[0144] Node E: predicted execution time is 15 minutes;
[0145] Node F: The predicted execution time is 30 minutes.
[0146] Select the target working node:
[0147] According to the predicted execution time, the node E with the shortest execution time is selected as the target working node.
[0148] Task allocation: The system allocates the task to node E and starts the execution of the task. The task start time is the current time and the estimated completion time is 15 minutes later.
[0149] Through the above steps, this embodiment can efficiently identify the task information of the task scheduling request, obtain the work information of the working node, use the LASSO regression prediction algorithm to predict the task execution time, and allocate tasks according to the execution time, thereby improving the overall efficiency and resource utilization of the system.
[0150] Furthermore, a third embodiment of the intelligent task scheduling method of the present invention is proposed. In the above step a1, after the step of identifying the task information of the task scheduling request, the method further includes:
[0151] Step b1, cleaning and converting the task information and the work information, removing data irrelevant to the task to be performed, and standardizing or normalizing the task information and the work information to meet the input requirements of the LASSO regression prediction algorithm.
[0152] In this embodiment, the task information and the work information of the work node are cleaned and converted, data irrelevant to the task to be executed is removed, and the data is standardized or normalized to meet the input requirements of the LASSO regression prediction algorithm. The specific steps may include:
[0153] Data cleaning: Remove irrelevant or abnormal data in task and node information, such as invalid or missing values.
[0154] Data conversion: Convert the original data into a format suitable for the LASSO regression prediction algorithm.
[0155] Data normalization: Convert numerical data to the same dimension to ensure that different features can be compared on the same scale.
[0156] Data normalization: Scaling data to a specific range (such as 0 to 1) to improve the performance and stability of the algorithm.
[0157] By cleaning and converting the data, the system in this embodiment can better utilize these high-quality, standardized data to predict task execution time and schedule resources, thereby improving the performance and efficiency of the overall system.
[0158] Furthermore, a fourth embodiment of the intelligent task scheduling method of the present invention is proposed. In the above step S10, the step of registering the working node to the registration center includes:
[0159] Step c1, determining whether the working node is available;
[0160] Step c2: If yes, register the available working nodes to the registration center, and record the general information of the working nodes through the registration center, wherein the general information includes the node type of the working nodes.
[0161] In this embodiment, before registering the working node to the registration center, the system first determines whether the working node is online and available. The system can detect the network connection status and health status of the working node through the Ping command or API request to ensure that the node can operate normally.
[0162] If the working node is determined to be available, the system registers it with the registration center and records its general information. The system can send the general information of the working node to the registration interface of the registration center through a network protocol (such as HTTP or gRPC).
[0163] After receiving the information of the working node, the registration center stores it in the database. Common information includes node type, IP address, HOST name, etc. For example, the registration center parses the received information and stores it in a relational database or a NoSQL database for subsequent query and use.
[0164] Example description:
[0165] Determine whether the working node is available:
[0166] The system sends a Ping command to the working node A to check its network connection status. If node A responds to the Ping request, the system determines that it is online and available.
[0167] Register available working nodes to the registration center:
[0168] After node A is determined to be available, the system packages its general information (node type, IP address, HOST name) and sends it to the registration interface of the registration center through an HTTP POST request.
[0169] General information includes:
[0170] Node type: Application node
[0171] IP address: 192.168.1.10
[0172] HOST name: app-node-A
[0173] Records general information about the worker node.
[0174] After receiving the information of node A, the registration center parses the data packet and stores the information in the database. The node type is recorded as application node, the IP address is 192.168.1.10, and the host name is app-node-A.
[0175] Through the above steps, this embodiment can effectively determine the availability of working nodes, register the available working nodes to the registration center, and record their general information, thereby ensuring the accuracy and efficiency of task scheduling and resource management.
[0176] Furthermore, a fifth embodiment of the intelligent task scheduling method of the present invention is proposed. In the above step S30, the step of allocating a target working node for the task to be executed according to the execution time of the working node includes:
[0177] Step d1, determining a candidate working node that can execute the task according to the general information of the working node;
[0178] Step d2: taking the candidate working node with the shortest execution time as the target working node of the task to be executed.
[0179] In this embodiment, the system selects candidate working nodes that meet the task execution conditions based on the general information of the working nodes (such as node type, location, resource configuration, etc.). The general information of the working nodes in the registration center can be queried, and the nodes that meet the conditions can be filtered out as candidate nodes based on the specific requirements of the task (such as resource requirements, node type, etc.).
[0180] For each candidate work node, the LASSO regression prediction algorithm is used to predict the execution time of the task on the node based on the node's work information (such as CPU usage, memory usage, task queue length, etc.). For example, the latest work information of the candidate node is obtained and input into the trained LASSO regression model to obtain the execution time of the task on each candidate node. Alternatively, the execution time of all work nodes is calculated first, and then the candidate work nodes that can be used to execute the task are determined.
[0181] Compare the predicted execution time of each candidate node, select the node with the shortest execution time as the target working node, and assign the task to be executed to this node. The purpose is to select the optimal node, send the task assignment information to this node, and start the task execution.
[0182] Example description:
[0183] Assuming there is a data processing task, you can first determine the candidate working nodes:
[0184] The user submitted a data processing task that required 2 CPUs and 4 GB of memory.
[0185] The system queries the registration center and selects the candidate working nodes that meet the conditions:
[0186] Node A: application node, with 8-core CPU and 16 GB memory.
[0187] Node B: application node, CPU configuration 4 cores, memory 8GB.
[0188] Node C: database node, does not meet the conditions and is excluded.
[0189] Predict execution time:
[0190] The system obtains the latest working information of nodes A and B:
[0191] Node A: CPU usage 70%, memory usage 65%, task queue length 3.
[0192] Node B: CPU usage 50%, memory usage 55%, task queue length 2.
[0193] Use the LASSO regression model to predict the execution time of the task on each node:
[0194] Node A: The prediction execution time is 15 minutes.
[0195] Node B: The prediction execution time is 10 minutes.
[0196] Select the target working node:
[0197] The system compares the predicted execution times and selects node B with the shortest execution time as the target working node.
[0198] Assign the task to node B. The task start time is the current time and the estimated completion time is 10 minutes later.
[0199] Through the above steps, this embodiment can efficiently determine the candidate working nodes that can execute the task based on the general information of the working nodes, use the LASSO regression algorithm to predict the execution time, and select the node with the shortest execution time as the target working node, thereby improving the overall efficiency and resource utilization of the system.
[0200] Furthermore, a sixth embodiment of the intelligent task scheduling method of the present invention is proposed, after the above step S30, it also includes:
[0201] Step e1, monitor the task execution status of the target working node and dynamically adjust task allocation according to task requirements.
[0202] In this embodiment, the task execution status of the target working node is monitored in real time, including resource usage, task progress, etc. The working information of the target node is continuously acquired through the metadata collection component, and compared with the expected execution status to ensure that the task is carried out as planned.
[0203] Dynamically adjust task allocation based on abnormal situations or changes in task requirements found during monitoring. For example, when the target node is overloaded or fails, migrate the task to other candidate nodes. Based on real-time monitoring data, the decision center determines whether task allocation needs to be adjusted and performs task migration operations through the internal communication mechanism.
[0204] Example description:
[0205] Assume there is a data processing task, determine the candidate working nodes:
[0206] The user submitted a data processing task that required 2 CPUs and 4 GB of memory.
[0207] The system queries the registration center and selects the candidate working nodes that meet the conditions:
[0208] Node A: application node, with 8-core CPU and 16 GB memory.
[0209] Node B: application node, CPU configuration 4 cores, memory 8GB.
[0210] Node C: database node, does not meet the conditions and is excluded.
[0211] Predict execution time: The system obtains the latest work information of node A and node B:
[0212] Node A: CPU usage 70%, memory usage 65%, task queue length 3.
[0213] Node B: CPU usage 50%, memory usage 55%, task queue length 2.
[0214] Use the LASSO regression model to predict the execution time of the task on each node:
[0215] Node A: The prediction execution time is 15 minutes.
[0216] Node B: The prediction execution time is 10 minutes.
[0217] Select the target working node: The system compares the predicted execution time and selects the node B with the shortest execution time as the target working node. The task is assigned to node B, with the task start time being the current time and the estimated completion time being 10 minutes later.
[0218] Monitor task execution status:
[0219] The metadata collection component monitors the resource usage and task execution progress of Node B in real time:
[0220] Resource usage: CPU usage 55%, memory usage 60%.
[0221] Mission progress: The mission is 50% complete.
[0222] Dynamically adjust task allocation:
[0223] During the monitoring process, it was found that the CPU usage of node B continued to rise to 85%, and the task progress was slow. The decision center determined that task migration was necessary and migrated the task to node A with lower load. The task status and progress information was transmitted to node A, and node A continued to execute the task to ensure that the task was completed on time.
[0224] Through the above steps, this embodiment performs real-time monitoring and dynamic adjustment during the task execution process to ensure that the task is completed on time and improve the overall efficiency and resource utilization of the system.
[0225] Furthermore, a seventh embodiment of the intelligent task scheduling method of the present invention is proposed. In the above step S30, the step of allocating a target working node for the task to be executed according to the execution time of the working node includes:
[0226] Step f1, using a load balancing algorithm to determine the load status of the working node, and assigning a target working node to the task to be executed according to the load status and execution time of the working node.
[0227] In this embodiment, the specific steps may include:
[0228] Determine candidate work nodes: Based on the general information of the work nodes (such as node type, location, resource configuration, etc.), the system screens out candidate work nodes that meet the task execution conditions.
[0229] Use load balancing algorithms to determine load status: Use load balancing algorithms (such as polling algorithms, least connection algorithms, weighted polling algorithms, etc.) to evaluate the load status of each candidate node. Obtain the current load status of each candidate node, such as CPU usage, memory usage, task queue length, etc., and calculate the comprehensive load index of each node. The formula for the comprehensive load index can be:
[0230] Comprehensive load index = α·CPU usage + β·memory usage + γ·task queue length
[0231] Among them, α, β, and γ are weight coefficients, which are set according to specific needs to ensure that the influence of different indicators is balanced. CPU usage, memory usage, and task queue length are real-time performance indicators of each working node.
[0232] Predict execution time: For each candidate work node, use the LASSO regression prediction algorithm to predict the execution time of the task on the node based on the node's work information. Obtain the latest work information of the candidate node and input it into the trained LASSO regression model to obtain the execution time of the task on each candidate node.
[0233] Select the target working node: Consider the load status and predicted execution time of the candidate nodes, and select the most suitable node as the target working node. Compare the load status and execution time of each candidate node, select the node with lower load and shorter execution time as the target node, and assign the task to the node. The comprehensive load index and predicted execution time jointly determine the target node for task assignment. The total comprehensive index formula can be: total comprehensive index = comprehensive load index + predicted execution time.
[0234] Task allocation: Assign tasks to target working nodes, start task execution, and monitor their progress. Send task allocation information to target nodes through internal communication mechanisms, and monitor task execution status in real time to ensure successful completion of tasks.
[0235] Monitor task execution status: Monitor the task execution status of the target working node in real time, including resource usage, task progress, etc. The metadata collection component continuously obtains the working information of the target node and compares it with the expected execution status to ensure that the task proceeds as planned.
[0236] Dynamically adjust task allocation: Dynamically adjust task allocation based on abnormal situations or changes in task requirements found during monitoring. For example, when the target node is overloaded or fails, migrate the task to other candidate nodes. Based on real-time monitoring data, the decision center determines whether task allocation needs to be adjusted and performs task migration operations through the internal communication mechanism.
[0237] Example description:
[0238] Assume there is a data processing task, determine the candidate working nodes:
[0239] The user submits a data processing task, and the system queries the registration center to filter out candidate working nodes that meet the requirements:
[0240] Node A: CPU configuration 8 cores, memory 16 GB.
[0241] Node B: CPU configuration 4 cores, memory 8GB.
[0242] Use the load balancing algorithm to determine the load status:
[0243] Get the current load information of node A and node B:
[0244] Node A: CPU usage 70%, memory usage 65%, task queue length 3.
[0245] Node B: CPU usage 50%, memory usage 55%, task queue length 2.
[0246] Calculate the load using the composite load index formula:
[0247] Node A: The comprehensive load index is 0.5·0.7+0.3·0.65+0.2·0.3=0.605
[0248] Node B: The comprehensive load index is 0.5·0.5+0.3·0.55+0.2·0.2=0.455
[0249] Predict execution time:
[0250] Use the LASSO regression model to predict the execution time of the task on each node:
[0251] Node A: The prediction execution time is 15 minutes.
[0252] Node B: The prediction execution time is 10 minutes.
[0253] Comprehensive evaluation, calculate the total comprehensive index:
[0254] Node A: The total comprehensive index is 0.605+15=15.605.
[0255] Node B: The total comprehensive index is 0.455+10=10.455.
[0256] Select the target working node:
[0257] Node B is selected as the target working node because its total comprehensive index is lower.
[0258] Task Assignment:
[0259] Assign the task to node B. The task start time is the current time and the estimated completion time is 10 minutes later.
[0260] Through the above steps, this embodiment can effectively combine the load balancing algorithm and the execution time prediction, select the most suitable target working node for the task to be executed, and improve the overall efficiency and resource utilization of the system.
[0261] The invention also provides an intelligent task scheduling device.
[0262] Reference Figure 3 , Figure 3 This is a functional module diagram of the first embodiment of the intelligent task scheduling device of the present invention. The intelligent task scheduling device comprises:
[0263] A node registration module registers the working node to the registration center and obtains the working information of the working node in the registration center;
[0264] A task receiving module receives a task scheduling request for a task to be executed and sends the task scheduling request to a decision center;
[0265] The node allocation module determines the execution time of the working node according to the task scheduling request and the working information of the working node through the decision center, and allocates a target working node to the task to be executed according to the execution time of the working node.
[0266] In addition, an embodiment of the present invention further provides a storage medium.
[0267] The storage medium stores an intelligent task scheduling program, and when the intelligent task scheduling program is executed by the processor, the steps of the intelligent task scheduling method described above are implemented.
[0268] The specific implementation of the storage medium of the present invention is basically the same as the above-mentioned embodiments of the intelligent task scheduling method, and will not be repeated here.
[0269] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.
[0270] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0271] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk.
[0272] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0273] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. An intelligent task scheduling method, characterized in that: The intelligent task scheduling method comprises the following steps: Register the working node to the registration center and obtain the working information of the working node in the registration center; Receiving a task scheduling request for a task to be executed, and sending the task scheduling request to a decision center; The decision center determines the execution time of the working node according to the task scheduling request and the working information of the working node, and allocates a target working node to the task to be executed according to the execution time of the working node.
2. The intelligent task scheduling method according to claim 1, characterized in that: The step of determining the execution time of the working node according to the task scheduling request and the working information of the working node by the decision center, and allocating a target working node for the task to be executed according to the execution time of the working node comprises: Identify the task information of the task scheduling request, and use the LASSO regression prediction algorithm through the decision center to predict the execution time of the task to be executed on each working node according to the task information and the working information of the working node; The working node with the shortest execution time is selected as the target working node of the task to be executed.
3. The intelligent task scheduling method as claimed in claim 2, characterized in that: After the step of identifying the task information of the task scheduling request, the method further includes: The task information and the work information are cleaned and converted to remove data irrelevant to the task to be performed, and the task information and the work information are standardized or normalized to meet the input requirements of the LASSO regression prediction algorithm.
4. The intelligent task scheduling method according to claim 1, characterized in that: The step of registering the working node to the registration center includes: Determine whether the working node is available; If so, the available working nodes are registered with a registration center, and general information of the working nodes is recorded by the registration center, where the general information includes the node type of the working node.
5. The intelligent task scheduling method according to claim 4, characterized in that: The step of allocating a target working node for the task to be executed according to the execution time of the working node comprises: Determine a candidate working node that can execute the task according to the general information of the working node; The candidate working node with the shortest execution time is used as the target working node of the task to be executed.
6. The intelligent task scheduling method according to claim 1, characterized in that: After the step of allocating a target working node for the task to be executed according to the execution time of the working node, the method further includes: Monitor the task execution status of the target working node and dynamically adjust task allocation according to task requirements.
7. The intelligent task scheduling method according to claim 1, characterized in that: The step of allocating a target working node for the task to be executed according to the execution time of the working node comprises: The load status of the working node is determined by using a load balancing algorithm, and a target working node is allocated to the task to be executed according to the load status and execution time of the working node.
8. An intelligent task scheduling device, characterized in that: The intelligent task scheduling device comprises: A node registration module registers the working node to the registration center and obtains the working information of the working node in the registration center; A task receiving module receives a task scheduling request for a task to be executed and sends the task scheduling request to a decision center; The node allocation module determines the execution time of the working node according to the task scheduling request and the working information of the working node through the decision center, and allocates a target working node to the task to be executed according to the execution time of the working node.
9. An intelligent task scheduling device, characterized in that: The intelligent task scheduling device includes a memory, a processor, and an intelligent task scheduling program stored in the memory and executable on the processor. When the intelligent task scheduling program is executed by the processor, the steps of the intelligent task scheduling method according to any one of claims 1 to 7 are implemented.
10. A storage medium, characterized in that: The storage medium stores an intelligent task scheduling program, and when the intelligent task scheduling program is executed by the processor, the steps of the intelligent task scheduling method according to any one of claims 1 to 7 are implemented.
