Task scheduling method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a task scheduling method and device, computer equipment, a computer readable storage medium and a computer program product. The method is used for a central node in a distributed system, and comprises the following steps: acquiring a task to be processed; according to a task demand corresponding to the to-be-processed task and a network performance parameter corresponding to each execution node in the distributed system, determining a target execution node from each execution node; and distributing the to-be-processed task to the target execution node for task processing. By adopting the method, the network performance in the high-throughput computing system can be monitored and optimized.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a task scheduling method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In a distributed system, a high-throughput computing system can be used for data processing. Among them, high-throughput computing (HTC) is a computing mode commonly used in large-scale scientific computing. It decomposes a large computing task into multiple subtasks, clarifies the resource requirements of each subtask, and then submits the requirements to the central node. At the same time, the central node collects available resources in the resource pool, matches the available resources with the requirements of each subtask, and distributes tasks, manages and schedules computing resources. With the increase in the scale of computing tasks and the complexity of the network environment, high-throughput computing faces the problem of network performance bottleneck when processing computing tasks.

[0003] However, existing high-throughput computing systems mainly focus on the management and scheduling of computing resources, and lack monitoring and optimization of network performance. Summary of the invention

[0004] Embodiments of the present application provide a task scheduling method, apparatus, computer equipment, computer-readable storage medium, and computer program product, which can monitor and optimize network performance in a high-throughput computing system.

[0005] In a first aspect, the present application provides a task scheduling method, the method comprising:

[0006] Get pending tasks;

[0007] Determining a target execution node from each of the execution nodes according to a task requirement corresponding to the task to be processed and a network performance parameter corresponding to each of the execution nodes in the distributed system;

[0008] The tasks to be processed are distributed to the target execution nodes for task processing.

[0009] In one embodiment, determining a target execution node from each of the execution nodes according to a task requirement corresponding to the task to be processed and a network performance parameter corresponding to each of the execution nodes in the distributed system includes:

[0010] According to the task requirement, determining at least one candidate execution node that meets the task requirement from each of the execution nodes;

[0011] The target network performance parameters corresponding to each of the candidate execution nodes are determined from each of the network performance parameters, and the target execution node is determined from each of the candidate execution nodes according to each of the target network performance parameters.

[0012] In one embodiment, determining the target execution node from the candidate execution nodes according to the target network performance parameters includes:

[0013] For each of the candidate execution nodes, determining a task allocation priority corresponding to the candidate execution node according to the target network performance parameter;

[0014] The candidate execution node corresponding to the highest task allocation priority among the task allocation priorities is determined as the target execution node.

[0015] In one of the embodiments, determining the task allocation priority corresponding to the candidate execution node according to the target network performance parameter includes:

[0016] Acquire a physical resource parameter corresponding to the candidate execution node, and determine a performance quantization value corresponding to the candidate execution node according to the target network performance parameter and the physical resource parameter;

[0017] The task allocation priority corresponding to the candidate execution node is determined according to the performance quantization value, and the magnitude of the performance quantization value is positively correlated with the priority of the task allocation priority.

[0018] In one embodiment, the method further comprises:

[0019] Receive the network performance parameters corresponding to each execution node sent by each execution node.

[0020] In a second aspect, the present application also provides a task scheduling method, the method comprising:

[0021] Receiving a task to be processed assigned by a central node in the distributed system, wherein the target execution node is determined by the central node from each execution node according to a task requirement corresponding to the task to be processed and a network performance parameter corresponding to each execution node in the distributed system;

[0022] Perform task processing on the task to be processed.

[0023] In one embodiment, the target execution node is connected to a gateway device, and the method further includes:

[0024] Monitoring the network performance parameters corresponding to the target execution node through the gateway device according to a preset time period;

[0025] The network performance parameters are sent to the central node.

[0026] In a third aspect, the present application further provides a device for a central node in a distributed system, the device comprising:

[0027] The acquisition module is used to obtain tasks to be processed;

[0028] A determination module, configured to determine a target execution node from each of the execution nodes according to a task requirement corresponding to the task to be processed and a network performance parameter corresponding to each of the execution nodes in the distributed system;

[0029] The allocation module is used to allocate the tasks to be processed to the target execution node for task processing.

[0030] In a fourth aspect, the present application further provides a device for a target execution node in a distributed system, the device comprising:

[0031] A second receiving module is used to receive a task to be processed assigned by a central node in the distributed system, wherein the target execution node is determined by the central node from each execution node according to a task requirement corresponding to the task to be processed and a network performance parameter corresponding to each execution node in the distributed system;

[0032] A processing module is used to perform task processing on the tasks to be processed.

[0033] In a fifth aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first and second aspects above are implemented.

[0034] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first and second aspects above are implemented.

[0035] In a seventh aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in the first and second aspects above.

[0036] In the above-mentioned task scheduling method, apparatus, computer equipment, computer-readable storage medium and computer program product, the central node obtains the task to be processed, and then, according to the task requirements corresponding to the task to be processed and the network performance parameters corresponding to each execution node in the distributed system, determines the target execution node from each execution node, and then distributes the task to be processed to the target execution node for task processing. Since the central node can consider the task requirements of the task to be processed and the network performance parameters corresponding to each execution node in the process of determining the target execution node, it can dynamically adjust the scheduling strategy of each execution node according to the network performance parameters, thereby avoiding the waste of network resources, and then optimizing the utilization of network resources, determining the target execution node that meets the processing requirements of the task to be processed, and realizing the monitoring and optimization of network performance in high-throughput computing systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG1 is an application environment diagram of a task scheduling method in one embodiment;

[0038] FIG2 is a schematic diagram of a process flow of a task scheduling method in one embodiment;

[0039] FIG3 is a schematic diagram of a flow chart of step 202 in another embodiment;

[0040] FIG4 is a schematic diagram of a flow chart of step 301 in another embodiment;

[0041] FIG5 is a schematic diagram of a flow chart of step 401 in another embodiment;

[0042] FIG6 is a schematic diagram of a flow chart of a task scheduling method in another embodiment;

[0043] FIG7 is a schematic diagram of a task scheduling method in another embodiment;

[0044] FIG8 is a structural block diagram of a task scheduling device in one embodiment;

[0045] FIG9 is a structural block diagram of a task scheduling device in another embodiment;

[0046] FIG. 10 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] The task scheduling method provided in the embodiment of the present application can be applied to the implementation environment as shown in Figure 1, wherein the central node 102 communicates with the execution node 104 through the network, and the execution node 104 may include multiple execution nodes. The first data storage system can store the data that the central node 102 needs to process, and the second data storage system can store the data that each execution node 104 needs to process. The first data storage system can be integrated in the central node 102, or it can be placed on the cloud or other network servers, and the second data storage system can be integrated in the execution node 104, or it can be placed on the cloud or other network servers. In the embodiment of the present application, the central node 102 can be a server or a virtual machine, and the execution node 104 can be a server or a virtual machine, and this embodiment does not limit this. Among them, after the central node obtains the task to be processed, according to the task requirements corresponding to the task to be processed and the network performance parameters corresponding to each execution node, the target execution node is determined from each execution node, and the task to be processed is assigned to the target execution node for task processing.

[0049] In traditional technology, in the high-throughput computing mode for distributed systems, when the central node allocates tasks to each execution node based on available resources, it only considers the matching relationship between available resources and task requirements, and does not consider the network performance parameters corresponding to each execution node. If the amount of data to be transmitted is relatively large and the network performance is poor, it may affect the overall performance of the distributed system. Based on this, there is a problem in traditional technology that it is impossible to monitor and optimize the network performance in high-throughput computing.

[0050] Based on the above-mentioned traditional technology, an embodiment of the present application provides a task scheduling method, which obtains the task to be processed, and then determines the target execution node from each execution node according to the task requirements corresponding to the task to be processed and the network performance parameters corresponding to each execution node in the distributed system. Then, the task to be processed is assigned to the target execution node for task processing. This method realizes dynamic adjustment of the scheduling strategy of each execution node, avoids waste of network resources, optimizes the utilization of network resources, determines the target execution node that meets the processing requirements of the task to be processed, and realizes the effect of monitoring and optimizing the network performance in the high-throughput computing system.

[0051] It should be noted that the beneficial effects brought about by the embodiments of the present application or the technical problems solved are not limited to this one, but may also include other implicit or related problems. For details, please refer to the description of the following embodiments.

[0052] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0053] In one embodiment, as shown in FIG2 , a task scheduling method is provided, which is described by taking the method applied to the central node in FIG1 as an example, and includes the following steps:

[0054] Step 201, obtaining tasks to be processed.

[0055] It should be noted that in the computing architecture of the high-throughput computing system applied to the distributed system, the central node in the high-throughput computing system is used to manage and schedule the available resources in the distributed system, and the execution node in the high-throughput computing system is used to monitor the network performance parameters and send the network performance parameters to the central node, so that the central node can allocate tasks to each execution node for task processing based on the network performance parameters and available resources.

[0056] Among them, pending tasks refer to tasks to be processed in a distributed system, for example, pending tasks may be tasks such as big data analysis, machine learning, pattern recognition, etc. It is understandable that the client can send the processing task to the central node, and the central node can immediately process the processing task, or, if the computing resources in the distributed system cannot meet the task requirements of the processing task, the central node can add the processing task to a preset task queue, so that when the computing resources can meet the task requirements of the processing task, the processing task is determined as a pending task.

[0057] Optionally, in this embodiment, the central node may receive the to-be-processed tasks sent by the client in real time through the network, or the central node may also obtain the to-be-processed tasks from the task queue.

[0058] Step 202 , determining a target execution node from among the execution nodes according to the task requirements corresponding to the task to be processed and the network performance parameters corresponding to the execution nodes in the distributed system.

[0059] Among them, task requirements refer to the computing resources required to process the tasks to be processed. For example, task requirements may include CPU resources, memory resources, hard disk resources, etc. Network performance parameters refer to parameters corresponding to data transmission through the network. Network performance can be analyzed based on network performance parameters, so that network configuration can be optimized according to the results of network performance analysis to improve network utilization. For example, network performance parameters may include bandwidth utilization, network delay, packet loss rate, etc. It can be understood that in a distributed system, the central node can transmit data to multiple execution nodes respectively. In this embodiment, the execution node used to execute the task to be processed is determined as the target execution node.

[0060] In this embodiment, the central node can determine the execution node that can meet the task requirements from each execution node based on the task requirements of the task to be processed. At the same time, the central node can determine the execution node that can meet the data transmission requirements of the task to be processed based on the network performance parameters corresponding to each execution node, thereby determining the execution node that can meet the task requirements and the data transmission requirements as the target execution node.

[0061] Step 203: Allocate the task to be processed to the target execution node for task processing.

[0062] In this embodiment, after the central node determines the target execution node from multiple execution nodes, it distributes the task to be processed to the target execution node through the network, so that the target execution node processes the task to be processed.

[0063] In the above-mentioned task scheduling method, the central node obtains the task to be processed, and then determines the target execution node from each execution node according to the task requirements corresponding to the task to be processed and the network performance parameters corresponding to each execution node in the distributed system. Then, the task to be processed is assigned to the target execution node for task processing. Since the central node can consider the task requirements of the task to be processed while also considering the network performance parameters corresponding to each execution node in the process of determining the target execution node, it can dynamically adjust the scheduling strategy of each execution node according to the network performance parameters, thereby avoiding the waste of network resources, and then optimizing the utilization of network resources, determining the target execution node that meets the processing requirements of the task to be processed, and realizing the monitoring and optimization of network performance in high-throughput computing systems.

[0064] In addition, in this task scheduling method, the target execution node is determined from each execution node according to the task requirements corresponding to the task to be processed and the network performance parameters corresponding to each execution node in the distributed system, thereby improving the reliability and processing efficiency of the determined target execution node, and also improving the accuracy of the central node in determining the target execution node, thereby improving the overall system performance of the high-throughput computing system.

[0065] Based on the embodiment shown in FIG. 2 above, referring to FIG. 3, this embodiment relates to a process of how a central node determines a target execution node from each execution node according to the task requirements corresponding to the task to be processed and the network performance parameters corresponding to each execution node in the distributed system. In one embodiment, as shown in FIG. 3, step 202 may include steps 301 and 302 shown in FIG. 3:

[0066] Step 301: According to the task requirement, at least one candidate execution node that meets the task requirement is determined from among the execution nodes.

[0067] Among them, the candidate execution node refers to an execution node that can meet the task requirements and needs to further determine whether it can process the task to be processed based on network performance parameters. It should be noted that in an embodiment of the present application, a distributed system may include a resource pool, which is used to manage the resources of each execution node, and the resources of each execution node can be divided into unavailable resources and available resources. The central node can manage and schedule the resources of each execution node based on the resource pool.

[0068] In this embodiment, after obtaining the tasks to be processed, the central node determines the task requirements corresponding to the tasks to be processed by analyzing the tasks to be processed. At the same time, it can send a resource acquisition instruction to the resource pool, and then receive the resources corresponding to each execution node sent by the resource pool in response to the resource acquisition instruction, thereby determining the available resources corresponding to each execution node. Further, the available resources corresponding to each execution node are compared with the task requirements. If the comparison results show that the available resources meet the task requirements, the execution nodes that meet the task requirements can be determined as candidate nodes.

[0069] Optionally, if there are multiple execution nodes that meet the task requirements among the execution nodes, the candidate nodes include multiple execution nodes; if there is one execution node that meets the task requirements among the execution nodes, the candidate nodes include one execution node.

[0070] For example, if the task requirement of the task to be processed is to occupy a memory space of 10M, the central node may determine the execution nodes whose memory space in the available resources is greater than or equal to 10M as candidate nodes.

[0071] Step 302: determine the target network performance parameters corresponding to each candidate execution node from each network performance parameter, and determine the target execution node from each candidate execution node according to each target network performance parameter.

[0072] It should be noted that after the central node determines the candidate execution nodes, it can further determine the target execution node based on the network performance parameters corresponding to the candidate execution nodes. In an embodiment of the present application, the central node can periodically receive the network performance parameters sent by each execution node. Therefore, when determining the target execution node, in order to save computing resources, the network performance parameters of each execution node may not be analyzed, but the network performance parameters corresponding to the candidate execution nodes may be analyzed, so as to determine the target execution node from each candidate execution node based on the analysis results. In this embodiment, the network performance parameters corresponding to each candidate execution node can be determined as the target network performance parameters.

[0073] It is understandable that the target network performance parameter corresponding to the candidate execution node can be determined as the target execution node only when it satisfies the data transmission. In this embodiment, the network performance parameter threshold can be set in advance according to the data transmission requirements, and then when the target network performance parameter is greater than the network performance parameter threshold, the candidate execution node corresponding to the target network performance parameter is determined as the target execution node.

[0074] In this embodiment, the central node can filter out the target network performance parameters corresponding to each candidate execution node from various network performance parameters according to the identifier corresponding to each candidate execution node, and then compare each target network performance parameter with a preset network performance parameter threshold. If a target network performance parameter is greater than the network performance parameter threshold, the candidate execution node corresponding to the target network performance can be determined as the target execution node.

[0075] In this embodiment, the central node first determines at least one candidate execution node that meets the task requirements from each execution node according to the task requirements, and then determines the target network performance parameters corresponding to each candidate execution node from each network performance parameter, and determines the target execution node from each candidate execution node according to each target network performance parameter. In the process of determining the target execution node, the candidate execution node is first determined according to the task requirements, and then the target execution node is determined according to the target network performance parameters corresponding to each candidate execution node. Compared with analyzing the network performance parameters of each execution node, the amount of data analyzed can be reduced, so that the target execution node can be determined quickly, thereby improving the efficiency of determining the target execution node.

[0076] Based on the embodiment shown in FIG. 3 above, referring to FIG. 4, this embodiment relates to a process of how the central node determines a target execution node from each candidate execution node according to each target network performance parameter. In one embodiment, as shown in FIG. 4, step 302 may include steps 401 and 402 shown in FIG. 4:

[0077] Step 401 : for each candidate execution node, determine the task allocation priority corresponding to the candidate execution node according to the target network performance parameter.

[0078] Among them, the task allocation priority refers to the priority of allocating pending tasks to candidate execution nodes for processing. It can be understood that the higher the task allocation priority of a candidate execution node, the more the central node will prioritize allocating pending tasks to the candidate execution node for execution.

[0079] In this embodiment, a candidate execution node may be taken as an example to describe the process of the central node determining the task allocation priority corresponding to the candidate execution node according to the target network performance parameter. The central node may compare the target network performance parameter corresponding to the candidate execution node with the target network performance parameters corresponding to other candidate execution nodes, and then evaluate the priority corresponding to the candidate execution node according to the comparison result, and then determine the task allocation priority corresponding to the candidate execution node according to the evaluation result.

[0080] Step 402: determine the candidate execution node corresponding to the highest task allocation priority among the task allocation priorities as the target execution node.

[0081] It can be understood that if the task allocation priority corresponding to a candidate execution node is higher, it means that the network performance of the candidate execution node is better.

[0082] In this embodiment, the central node can sort the task allocation priorities of the candidate execution nodes in order, then determine the task allocation priority that comes first in the order as the highest task allocation priority, and determine the candidate execution node corresponding to the highest task allocation priority as the target execution node.

[0083] In this embodiment, for each candidate execution node, the central node can determine the task allocation priority corresponding to the candidate execution node based on the target network performance parameters. Since the task allocation priority is determined based on the target network performance parameters, the higher the task allocation priority, the better the target network performance parameters. Furthermore, the candidate execution node corresponding to the highest task allocation priority is determined as the target execution node, so that the determined target execution node has the best network performance parameters among all candidate execution nodes, thereby allocating the tasks to be processed to the execution nodes with the best network performance parameters for processing, ensuring that the tasks to be processed can be processed under the best network conditions, thereby improving the processing efficiency of the tasks to be processed.

[0084] Based on the embodiment shown in FIG. 4 above, referring to FIG. 5, this embodiment relates to a process of how the central node determines the task allocation priority corresponding to the candidate execution node according to the target network performance parameter. In one embodiment, as shown in FIG. 5, step 401 may include steps 501 and 502 shown in FIG. 5:

[0085] Step 501: obtain physical resource parameters corresponding to the candidate execution node, and determine the performance quantization value corresponding to the candidate execution node according to the target network performance parameter and the physical resource parameter.

[0086] The physical resource parameters refer to the parameters of the physical device corresponding to the execution node. For example, the physical resource parameters may be the parameters of the server, or the physical resource parameters may be the parameters corresponding to the computer device where the virtual machine is located. It should be noted that the physical resource parameters are the parameters that have been determined when the execution node device is deployed, and are the maximum performance parameters corresponding to each execution node.

[0087] Optionally, the physical resource parameters may include CPU, memory, hard disk, etc. For example, CPU parameters may include the number of CPU cores, number of threads, basic frequency, etc., memory parameters may include memory capacity, type, etc., and hard disk parameters may include capacity, type, rotation speed, etc. It should be noted that for the physical resource parameters and target network performance parameters corresponding to the candidate execution nodes, the corresponding quantized values, i.e., performance quantized values, may be obtained using a preset calculation formula.

[0088] In this embodiment, for a candidate execution node, the central node can obtain the physical resource parameters corresponding to the candidate execution node from the pre-stored parameter records, and then bring the target network performance parameters and physical resource parameters corresponding to the candidate execution node into the preset calculation formula, and then determine the calculation result as the performance quantization value corresponding to the candidate execution node.

[0089] Step 502: determine the task allocation priority corresponding to the candidate execution node according to the performance quantization value, and the magnitude of the performance quantization value is positively correlated with the priority of the task allocation priority.

[0090] It can be understood that the performance quantization value can represent the performance corresponding to each candidate execution node. For example, the larger the performance quantization value, the better the performance of the candidate execution node, and the smaller the performance quantization value, the worse the performance of the candidate execution node. Therefore, the size of the performance quantization value is positively correlated with the priority of the task allocation priority, that is, the larger the performance quantization value, the higher the task allocation priority.

[0091] In this embodiment, the central node can use the lowest performance quantization value as the reference value to normalize the performance quantization value of the candidate execution node, thereby determining the task allocation priority corresponding to the candidate execution node according to the normalized result. For example, if the candidate execution node includes three execution nodes, namely, execution node A, execution node B, and execution node C, the normalized performance quantization value corresponding to execution node A is 0.2, the normalized performance quantization value corresponding to execution node B is 0.7, and the normalized performance quantization value corresponding to execution node C is 0.5, then the order of the task allocation priority corresponding to each candidate execution node can be determined as follows: execution node B>execution node C>execution node A.

[0092] In this embodiment, the central node can determine the performance quantization value corresponding to the candidate execution node according to the target network performance parameters and the physical resource parameters by obtaining the physical resource parameters corresponding to the candidate execution node. Since the performance quantization value takes into account both the physical resource parameters and the network performance parameters, the accuracy of the obtained performance quantization value can be improved, thereby determining the task allocation priority corresponding to the candidate execution node according to the performance quantization value. Since the size of the performance quantization value is positively correlated with the priority of the task allocation priority, the central node can quickly and accurately determine the task allocation priority corresponding to the candidate execution node according to the performance quantization value, thereby improving the efficiency and accuracy of the determined task allocation priority.

[0093] Based on the above embodiment, in one embodiment, the above method may further include: receiving network performance parameters corresponding to each execution node sent by each execution node.

[0094] It is understandable that in order for the central node to be able to analyze the network performance in a timely manner based on the network performance parameters sent by the execution nodes to ensure the stable operation of the network, the execution nodes can monitor their corresponding network performance parameters and send them to the central node.

[0095] Optionally, in this embodiment, the central node may receive network performance parameters corresponding to each execution node periodically sent by each execution node; or, the central node may also receive network performance parameters corresponding to each execution node sent by each execution node in real time.

[0096] In this embodiment, the central node receives the network performance parameters corresponding to each execution node sent by each execution node, and can determine the target execution node in time according to the network performance parameters when obtaining the task to be processed, so as to respond to the task to be processed in time and improve the processing efficiency of the task to be processed.

[0097] In one embodiment, as shown in FIG. 6 , a task scheduling method is further provided, which is described by taking an execution node in FIG. 1 as a target execution node as an example, and may include the following steps:

[0098] Step 601, receiving a task to be processed assigned by a central node in a distributed system, wherein a target execution node is determined by the central node from among the execution nodes according to a task requirement corresponding to the task to be processed and network performance parameters corresponding to each execution node in the distributed system.

[0099] It should be noted that the central node can determine the candidate execution node that meets the task requirements from each execution node based on the task requirements corresponding to the task to be processed, and determine the candidate execution node with a network status that can process the task to be processed based on the network performance parameters corresponding to each candidate execution node, thereby determining the candidate execution node as the target execution node, and then sending the task to be processed to the target execution node for processing.

[0100] In this embodiment, the target execution node can receive the to-be-processed task assigned by the central node through the network.

[0101] As a possible implementation mode, if the central node obtains multiple tasks to be processed, the target execution node corresponding to each task to be processed can be determined in parallel, and each task to be processed can be sent in parallel to each corresponding target execution node.

[0102] Step 602: Process the task to be processed.

[0103] It can be understood that the central node determines the candidate execution node corresponding to the highest task allocation priority as the target execution node. Therefore, after receiving the task to be processed, the target execution node can immediately process the task to be processed. In this embodiment, when the target execution node receives the task to be processed, it can call its own available resources to process the task to be processed.

[0104] In this embodiment, the target execution node can process the pending tasks by receiving the pending tasks assigned by the central node in the distributed system. Since the target execution node is determined by the central node from each execution node based on the task requirements corresponding to the pending tasks and the network performance parameters corresponding to each execution node in the distributed system, it ensures that the network status of the target execution node can meet the processing requirements of the pending tasks, thereby improving the processing efficiency of the pending tasks and further improving the overall performance of the distributed system.

[0105] In one embodiment, as shown in FIG. 7 , the target execution node is connected to a gateway device, and the method may further include:

[0106] Step 701: monitor network performance parameters corresponding to a target execution node through a gateway device according to a preset time period.

[0107] Among them, network performance parameters may include bandwidth utilization, network delay, packet loss rate, etc. For example, the gateway device can determine the network delay and bandwidth utilization by obtaining the duration of data transmission and reception, and determine whether there is packet loss based on the received response message, thereby calculating the packet loss rate.

[0108] In this embodiment, the gateway device in the target execution node can monitor the network performance parameters in real time, and the target execution node can obtain the monitored network performance parameters from the gateway device according to a preset time period.

[0109] As a possible implementation, the execution node may also obtain corresponding network performance parameters through software. For example, a network monitoring module may be set in each execution node, and the duration of data transmission and reception may be obtained by running the network monitoring module, so as to determine the network delay according to the duration of data transmission and reception, and the network performance parameters of the process.

[0110] Step 702: Send the network performance parameters to the central node.

[0111] It can be understood that in order to save transmission resources and enable the central node to timely analyze the network performance based on the network performance parameters sent by the execution node to ensure the stable operation of the network, in this embodiment, the execution node can periodically send the network performance parameters to the central node according to a time period.

[0112] As a possible implementation, the execution node may also send the monitored network performance parameters to the central node in real time.

[0113] In this embodiment, the target execution node monitors the network performance parameters corresponding to the target execution node through the gateway device according to a preset time period, and sends the network performance parameters to the central node, so that the central node can grasp the network status inside the system according to the network performance parameters, and dynamically adjust the task scheduling strategy according to the network performance parameters to improve the overall performance of the system.

[0114] To facilitate understanding by those skilled in the art, the task scheduling method provided by the present application is described in detail below by taking the method applied to a distributed system including a central node and a target execution node as an example. The method may include:

[0115] S1, according to a preset time period, the target execution node monitors the network performance parameters corresponding to the target execution node through the gateway device.

[0116] S2, the target execution node sends the network performance parameters to the central node.

[0117] S3, the central node obtains the tasks to be processed.

[0118] S4, the central node determines at least one candidate execution node that meets the task requirements from each execution node according to the task requirements.

[0119] S5, the central node determines the target network performance parameters corresponding to each candidate execution node from the network performance parameters.

[0120] S6, for each candidate execution node, the central node obtains the physical resource parameters corresponding to the candidate execution node, and determines the performance quantization value corresponding to the candidate execution node according to the target network performance parameters and the physical resource parameters.

[0121] S7, the central node determines the task allocation priority corresponding to the candidate execution node according to the performance quantization value, and the size of the performance quantization value is positively correlated with the priority of the task allocation priority.

[0122] S8, the central node determines the candidate execution node corresponding to the highest task allocation priority among the task allocation priorities as the target execution node.

[0123] S9, the central node distributes the tasks to be processed to the target execution nodes.

[0124] S10, the target execution node receives the task to be processed and processes the task to be processed.

[0125] It should be understood that, although the various steps in the above flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above flowchart may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0126] In one embodiment, as shown in FIG8 , a task scheduling device is provided, which is used in a central node of a distributed system, and includes: an acquisition module 801, a determination module 802 and an allocation module 803, wherein:

[0127] An acquisition module 801 is used to acquire tasks to be processed;

[0128] A determination module 802 is used to determine a target execution node from each execution node according to a task requirement corresponding to the task to be processed and a network performance parameter corresponding to each execution node in the distributed system;

[0129] The allocation module 803 is used to allocate the tasks to be processed to the target execution node for task processing.

[0130] In one embodiment, the determination module 802 includes: a first determination unit and a second determination unit, wherein:

[0131] A first determining unit, configured to determine, according to the task requirement, at least one candidate execution node that meets the task requirement from among the execution nodes;

[0132] The second determination unit is used to determine the target network performance parameters corresponding to each candidate execution node from each network performance parameter, and determine the target execution node from each candidate execution node according to each target network performance parameter.

[0133] In one of the embodiments, the second determining unit is specifically used to: determine, for each candidate execution node, a task allocation priority corresponding to the candidate execution node according to the target network performance parameter;

[0134] The candidate execution node corresponding to the highest task allocation priority among the task allocation priorities is determined as the target execution node.

[0135] In one of the embodiments, the second determination unit is specifically used to: obtain a physical resource parameter corresponding to the candidate execution node, and determine a performance quantization value corresponding to the candidate execution node according to the target network performance parameter and the physical resource parameter;

[0136] The task allocation priority corresponding to the candidate execution node is determined according to the performance quantization value, and the size of the performance quantization value is positively correlated with the priority of the task allocation priority.

[0137] In one embodiment, the above device further includes: a first receiving module, wherein:

[0138] The first receiving module is used to receive network performance parameters corresponding to each execution node sent by each execution node.

[0139] For the specific definition of the task scheduling device, please refer to the definition of the task scheduling method above, which will not be repeated here. Each module in the above-mentioned task scheduling device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0140] In one embodiment, as shown in FIG. 9 , a task scheduling device is further provided, which is used in a target execution node in a distributed system, and includes: a second receiving module 901 and a processing module 902, wherein:

[0141] The second receiving module 901 is used to receive a task to be processed assigned by a central node in a distributed system, wherein the target execution node is determined by the central node from each execution node according to a task requirement corresponding to the task to be processed and a network performance parameter corresponding to each execution node in the distributed system;

[0142] The processing module 902 is used to perform task processing on the tasks to be processed.

[0143] In one embodiment, the target execution node is connected to a gateway device, and the apparatus further comprises:

[0144] A monitoring module is used to monitor the network performance parameters corresponding to the target execution node through the gateway device according to a preset time period;

[0145] The sending module is used to send the network performance parameters to the central node.

[0146] For the specific definition of the task scheduling device, please refer to the definition of the task scheduling method above, which will not be repeated here. Each module in the above-mentioned task scheduling device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0147] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG10. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store network performance parameters. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a task scheduling method is implemented.

[0148] Those skilled in the art will understand that the structure shown in FIG. 10 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0149] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0150] Get pending tasks;

[0151] Determine a target execution node from each execution node according to the task requirements corresponding to the task to be processed and the network performance parameters corresponding to each execution node in the distributed system;

[0152] Assign pending tasks to the target execution node for task processing.

[0153] In one embodiment, the processor implements the following steps when executing the computer program:

[0154] According to the task requirements, determining at least one candidate execution node that meets the task requirements from each execution node;

[0155] A target network performance parameter corresponding to each candidate execution node is determined from each network performance parameter, and a target execution node is determined from each candidate execution node based on each target network performance parameter.

[0156] In one embodiment, the processor implements the following steps when executing the computer program:

[0157] For each candidate execution node, determine the task allocation priority corresponding to the candidate execution node according to the target network performance parameter;

[0158] The candidate execution node corresponding to the highest task allocation priority among the task allocation priorities is determined as the target execution node.

[0159] In one embodiment, the processor implements the following steps when executing the computer program:

[0160] Obtaining physical resource parameters corresponding to the candidate execution node, and determining the performance quantization value corresponding to the candidate execution node according to the target network performance parameter and the physical resource parameter;

[0161] The task allocation priority corresponding to the candidate execution node is determined according to the performance quantization value, and the size of the performance quantization value is positively correlated with the priority of the task allocation priority.

[0162] In one embodiment, the processor implements the following steps when executing the computer program:

[0163] Receive network performance parameters corresponding to each execution node sent by each execution node.

[0164] In an exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0165] Receiving a task to be processed assigned by a central node in a distributed system, wherein a target execution node is determined by the central node from each execution node according to a task requirement corresponding to the task to be processed and a network performance parameter corresponding to each execution node in the distributed system;

[0166] Perform task processing on pending tasks.

[0167] In one embodiment, the processor implements the following steps when executing the computer program:

[0168] According to the preset time period, the network performance parameters corresponding to the target execution node are monitored through the gateway device;

[0169] Send network performance parameters to the central node.

[0170] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0171] Get pending tasks;

[0172] Determine a target execution node from each execution node according to the task requirements corresponding to the task to be processed and the network performance parameters corresponding to each execution node in the distributed system;

[0173] Assign pending tasks to the target execution node for task processing.

[0174] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0175] According to the task requirements, determining at least one candidate execution node that meets the task requirements from each execution node;

[0176] A target network performance parameter corresponding to each candidate execution node is determined from each network performance parameter, and a target execution node is determined from each candidate execution node based on each target network performance parameter.

[0177] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0178] For each candidate execution node, determine the task allocation priority corresponding to the candidate execution node according to the target network performance parameter;

[0179] The candidate execution node corresponding to the highest task allocation priority among the task allocation priorities is determined as the target execution node.

[0180] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0181] Obtaining physical resource parameters corresponding to the candidate execution node, and determining the performance quantization value corresponding to the candidate execution node according to the target network performance parameter and the physical resource parameter;

[0182] The task allocation priority corresponding to the candidate execution node is determined according to the performance quantization value, and the size of the performance quantization value is positively correlated with the priority of the task allocation priority.

[0183] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0184] Receive network performance parameters corresponding to each execution node sent by each execution node.

[0185] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0186] Receiving a task to be processed assigned by a central node in a distributed system, wherein a target execution node is determined by the central node from each execution node according to a task requirement corresponding to the task to be processed and a network performance parameter corresponding to each execution node in the distributed system;

[0187] Perform task processing on pending tasks.

[0188] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0189] According to the preset time period, the network performance parameters corresponding to the target execution node are monitored through the gateway device;

[0190] Send network performance parameters to the central node.

[0191] The present application also provides a computer program product including instructions, which, when executed on a computer, causes the computer to perform the following steps:

[0192] Get pending tasks;

[0193] Determine a target execution node from each execution node according to the task requirements corresponding to the task to be processed and the network performance parameters corresponding to each execution node in the distributed system;

[0194] Assign pending tasks to the target execution node for task processing.

[0195] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0196] According to the task requirements, determining at least one candidate execution node that meets the task requirements from each execution node;

[0197] A target network performance parameter corresponding to each candidate execution node is determined from each network performance parameter, and a target execution node is determined from each candidate execution node based on each target network performance parameter.

[0198] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0199] For each candidate execution node, determine the task allocation priority corresponding to the candidate execution node according to the target network performance parameter;

[0200] The candidate execution node corresponding to the highest task allocation priority among the task allocation priorities is determined as the target execution node.

[0201] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0202] Obtaining physical resource parameters corresponding to the candidate execution node, and determining the performance quantization value corresponding to the candidate execution node according to the target network performance parameter and the physical resource parameter;

[0203] The task allocation priority corresponding to the candidate execution node is determined according to the performance quantization value, and the size of the performance quantization value is positively correlated with the priority of the task allocation priority.

[0204] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0205] Receive network performance parameters corresponding to each execution node sent by each execution node.

[0206] The present application also provides a computer program product including instructions, which, when executed on a computer, causes the computer to perform the following steps:

[0207] Receiving a task to be processed assigned by a central node in a distributed system, wherein a target execution node is determined by the central node from each execution node according to a task requirement corresponding to the task to be processed and a network performance parameter corresponding to each execution node in the distributed system;

[0208] Perform task processing on pending tasks.

[0209] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0210] According to the preset time period, the network performance parameters corresponding to the target execution node are monitored through the gateway device;

[0211] Send network performance parameters to the central node.

[0212] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0213] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0214] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A task scheduling method, characterized in that: For use in a central node in a distributed system, the method comprises: Get pending tasks; Determining a target execution node from each of the execution nodes according to a task requirement corresponding to the task to be processed and a network performance parameter corresponding to each of the execution nodes in the distributed system; The tasks to be processed are distributed to the target execution nodes for task processing.

2. The method according to claim 1, characterized in that The determining a target execution node from each of the execution nodes according to the task requirements corresponding to the task to be processed and the network performance parameters corresponding to each of the execution nodes in the distributed system includes: According to the task requirement, determining at least one candidate execution node that meets the task requirement from each of the execution nodes; The target network performance parameters corresponding to each of the candidate execution nodes are determined from each of the network performance parameters, and the target execution node is determined from each of the candidate execution nodes according to each of the target network performance parameters.

3. The method according to claim 2, characterized in that The step of determining the target execution node from the candidate execution nodes according to the target network performance parameters includes: For each of the candidate execution nodes, determining a task allocation priority corresponding to the candidate execution node according to the target network performance parameter; The candidate execution node corresponding to the highest task allocation priority among the task allocation priorities is determined as the target execution node.

4. The method according to claim 3, characterized in that The determining, according to the target network performance parameter, the task allocation priority corresponding to the candidate execution node includes: Acquire a physical resource parameter corresponding to the candidate execution node, and determine a performance quantization value corresponding to the candidate execution node according to the target network performance parameter and the physical resource parameter; The task allocation priority corresponding to the candidate execution node is determined according to the performance quantization value, and the magnitude of the performance quantization value is positively correlated with the priority of the task allocation priority.

5. The method according to claim 1, characterized in that The method further comprises: Receive the network performance parameters corresponding to each execution node sent by each execution node.

6. A task scheduling method, characterized in that: For a target execution node in a distributed system, the method comprises: Receiving a task to be processed assigned by a central node in the distributed system, wherein the target execution node is determined by the central node from each execution node according to a task requirement corresponding to the task to be processed and a network performance parameter corresponding to each execution node in the distributed system; Perform task processing on the task to be processed.

7. The method according to claim 6, characterized in that The target execution node is connected to a gateway device, and the method further comprises: Monitoring the network performance parameters corresponding to the target execution node through the gateway device according to a preset time period; The network performance parameters are sent to the central node.

8. A task scheduling device, characterized in that: A central node in a distributed system, the device comprising: The acquisition module is used to obtain tasks to be processed; A determination module, configured to determine a target execution node from each of the execution nodes according to a task requirement corresponding to the task to be processed and a network performance parameter corresponding to each of the execution nodes in the distributed system; The allocation module is used to allocate the tasks to be processed to the target execution node for task processing.

9. A task scheduling device, characterized in that: For a target execution node in a distributed system, the device comprises: A second receiving module is used to receive a task to be processed assigned by a central node in the distributed system, wherein the target execution node is determined by the central node from each execution node according to a task requirement corresponding to the task to be processed and a network performance parameter corresponding to each execution node in the distributed system; A processing module is used to perform task processing on the task to be processed.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images