Task scheduling method, distributed system and storage medium

Abstract

One or more embodiments of the present specification provide a task scheduling method, a distributed system and a storage medium, the method is applied to a distributed system, the distributed system includes multiple data nodes, and the data nodes include multiple NUMA nodes; the method includes: determining a first target data node according to the resources required by the task to be scheduled and the allocatable resources of each data node; for each first target data node, determining a target NUMA node suitable for the task to be scheduled to run according to the resources required by the task to be scheduled and the allocatable resources of each NUMA node; selecting a second target data node with a lower load from the first target data node according to the load of the target NUMA node in each first target data node; deploying the task to be scheduled to the target NUMA node in the second target data node. This embodiment is conducive to providing the running performance of the task to be scheduled on the data node.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of computer software technology, and in particular, to a task scheduling method, a distributed system, and a storage medium. Background Art

[0002] A distributed system is a loosely coupled system consisting of multiple data nodes interconnected by communication lines. Each data node is a computing device that can independently process a transaction. With the development of technology, in order to adapt to the gradually expanding scale of data processing, computing devices with multiple NUMA nodes are gradually being used in distributed systems.

[0003] NUMA (Non-Uniform Memory Access Architecture) is a memory architecture designed for multi-processor devices. A NUMA node contains a group of processors (such as multiple physical CPUs). This group of processors share the same local memory, and the memory is directly bound to the processor through the memory controller. Processors in multiple NUMA nodes can access all system physical memories, but the time required to access the memory in this NUMA node is much less than the time required to access the memory in other NUMA nodes.

[0004] Currently, such distributed systems of computing devices with multiple NUMA nodes have problems with unreasonable task scheduling that affects performance. Summary of the invention

[0005] In view of this, one or more embodiments of the present specification provide a task scheduling method, a distributed system, and a storage medium.

[0006] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0007] According to a first aspect of one or more embodiments of this specification, a task scheduling method is proposed, which is applied to a distributed system, wherein the distributed system includes a plurality of data nodes, and the data nodes include a plurality of NUMA nodes; the method includes:

[0008] According to the resources required by the task to be scheduled and the allocatable resources of each of the data nodes, determining at least one first target data node whose allocatable resources are greater than the required resources;

[0009] For each of the first target data nodes, determining one or more target NUMA nodes suitable for running the task to be scheduled according to the resources required by the task to be scheduled and the allocatable resources of each of the NUMA nodes;

[0010] Selecting a second target data node from the first target data nodes according to the load of one or more target NUMA nodes among the first target data nodes; the load of the second target data node is lower than at least half of the load of the first target data node;

[0011] Deploy the task to be scheduled to one or more target NUMA nodes in the second target data node.

[0012] According to a second aspect of one or more embodiments of this specification, a distributed system is provided, wherein the distributed system includes a plurality of data nodes, and the data nodes include one or more NUMA nodes;

[0013] The distributed system is used to execute the method described in the first aspect.

[0014] According to a third aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] One or more embodiments of the present specification provide a task scheduling method, a distributed system and a storage medium, firstly, screening is performed from the data node granularity, and according to the resources required for the task to be scheduled and the allocatable resources of each of the data nodes, the first target data node whose allocatable resources are greater than the required resources is determined; then screening is performed from the NUMA node granularity in the data node, and for the first target data node, according to the resources required for the task to be scheduled and the allocatable resources of each of the NUMA nodes, one or more target NUMA nodes suitable for the running of the task to be scheduled are determined, and then according to the load of one or more target NUMA nodes in the first target data node, the second target data node with lower load is selected from the first target data node; finally, after the second target data node is screened out, the task to be scheduled is deployed to one or more target NUMA nodes in the second target data node. This embodiment implements a more refined scheduling through a multi-layer screening process from the data node granularity to the NUMA node granularity, and performs scheduling based on the load situation perceived by the NUMA node granularity, and can globally screen out one or more target NUMA nodes with lower load that meet the task running, thereby improving the performance of task deployment on the data node. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a structural diagram of a distributed system provided by an exemplary embodiment.

[0017] Figure 2It is a schematic diagram of the structure of a data node with a NUMA node provided by an exemplary embodiment.

[0018] Figure 3 It is a schematic diagram of the structure of another distributed system provided by an exemplary embodiment.

[0019] Figure 4 It is a flowchart of a task scheduling method provided by an exemplary embodiment.

[0020] Figure 5 It is a schematic diagram of a data node with tasks deployed provided by an exemplary embodiment.

[0021] Figure 6 It is a structural diagram of a K8s cluster provided by an exemplary embodiment. DETAILED DESCRIPTION

[0022] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0023] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0024] With the development of technology, in order to adapt to the gradually expanding scale of data processing, computing devices with multiple NUMA nodes are gradually being used in distributed systems. Figure 1 A distributed system is a loosely coupled system consisting of multiple data nodes 100 interconnected by communication lines. Each data node 100 is a computing device that can independently process a transaction, and the computing device can be a physical device or a virtual machine.

[0025] NUMA, or Non-Uniform Memory Access Architecture, is a distributed memory access method. Processors (such as CPUs) can access different memory addresses at the same time, greatly improving parallelism. In multi-NUMA mode, a data node (a computing device in the physical machine dimension) includes multiple NUMA nodes, and a NUMA node contains a group of processors (such as multiple physical CPUs, each of which can have multiple cores or threads). Take the CPU as an example, see Figure 2 , a data node 100 includes multiple NUMA nodes 10, Figure 2 Where n represents an integer greater than 0; each NUMA node 10 includes a plurality of CPUs 11, Figure 2 In the example, there are three CPUs in a NUMA node. Multiple CPUs 11 included in NUMA node 10 share the same local memory 12. Multiple CPUs 11 in NUMA node 10 use a shared memory controller. Therefore, all memories of NUMA node 10 are equivalent for all CPUs 11 in this NUMA node, but are different for CPUs in other NUMA nodes.

[0026] The processors in all NUMA nodes can access all the system physical memories, but the time required to access the memory in this NUMA node is much less than the time required to access the memory in other NUMA nodes. NUMA nodes can be divided into three types: local node, neighbor node, and remote node: Local node: For all CPUs in a NUMA node, this NUMA node is called a local node. Neighbor node: A NUMA node adjacent to a local node is called a neighbor node. Remote node: A NUMA node that is not a local node or a neighbor node is called a remote node. Neighbor nodes and remote nodes are called off nodes. It can be understood that the local nodes, neighbor nodes, and remote nodes mentioned here refer to nodes in the NUMA dimension, not data nodes in the physical machine dimension. The speed at which the processor (CPU) accesses the memory of different types of NUMA nodes is different: local node > neighbor node > remote node. The speed of accessing the local node is the fastest, and the speed of accessing the remote node is the slowest, that is, the access speed is related to the distance of the NUMA node. The longer the distance, the slower the access speed.

[0027] This type of distributed system with computing devices with multiple NUMA nodes schedules tasks based on the load conditions perceived at the data node granularity. When tasks are deployed in NUMA nodes, processor problems (such as CPU switching, CPU drift, or CPU current limiting) or cross-NUMA node memory access problems are prone to occur. Obviously, unreasonable task scheduling affects performance.

[0028] In view of the problem that a distributed system of a computing device with multiple NUMA nodes in the related art has unreasonable task scheduling that affects performance, an embodiment of the present specification provides a task scheduling method, which firstly performs screening from the data node granularity, and determines a first target data node whose allocatable resources are greater than the required resources according to the resources required for the task to be scheduled and the allocatable resources of each of the data nodes; then performs screening from the NUMA node granularity in the data node, and for the first target data node, determines one or more target NUMA nodes suitable for the running of the task to be scheduled according to the resources required for the task to be scheduled and the allocatable resources of each of the NUMA nodes, and then selects a second target data node with a lower load from the first target data node according to the load of one or more target NUMA nodes in the first target data node; finally, after screening out the second target data node, the task to be scheduled is deployed to one or more target NUMA nodes in the second target data node. This embodiment implements more refined scheduling through a multi-layer screening process from data node granularity to NUMA node granularity, and performs scheduling based on the load conditions perceived at the NUMA node granularity. It can globally screen out one or more target NUMA nodes with lower loads that are suitable for task operation, thereby improving the performance of tasks deployed on data nodes.

[0029] In an exemplary application scenario, see Figure 3 , Figure 3 The structure diagram of a distributed system of computing devices with multiple NUMA nodes is shown. The distributed system includes multiple data nodes 100, each of which is a computing device that can independently process a certain transaction. The computing device can be a physical device or a virtual machine. The data node 100 includes multiple NUMA nodes 10. Figure 2 The NUMA node 10 includes a plurality of processors with shared memory. The processor may be, for example, a CPU (e.g. Figure 2As shown in the figure), or other types of processors, such as MCU (Microcontroller Unit), digital signal processor (Digital Signal Processor, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA), etc., can be specifically configured according to the actual application scenario.

[0030] In some embodiments, see Figure 4 , Figure 4 A flowchart of a task scheduling method is shown. The task scheduling method can be applied to Figure 3 In the distributed system shown, the distributed system includes a plurality of data nodes, and the data nodes include a plurality of NUMA nodes; the method includes:

[0031] In step S101, at least one first target data node whose allocatable resources are greater than the required resources is determined according to the resources required by the task to be scheduled and the allocatable resources of each of the data nodes.

[0032] In step S102, for each of the first target data nodes, one or more target NUMA nodes suitable for running the task to be scheduled are determined according to the resources required by the task to be scheduled and the allocatable resources of each of the NUMA nodes.

[0033] In step S103, a second target data node is selected from the first target data nodes according to the load of one or more target NUMA nodes in each of the first target data nodes; the load of the second target data node is lower than at least half of the load of the first target data node.

[0034] In step S104, the task to be scheduled is deployed to one or more target NUMA nodes in the second target data node.

[0035] This embodiment implements more refined scheduling through a multi-layer screening process from data node granularity to NUMA node granularity, and performs scheduling based on the load conditions perceived at the NUMA node granularity. It can globally screen out one or more target NUMA nodes with lower loads that are suitable for running the tasks to be scheduled, thereby improving the performance of the tasks to be scheduled deployed on the data nodes.

[0036] Among them, steps S101 to S104 can be executed by one of the data nodes in the distributed system (for the convenience of distinction, the data node is referred to as the control node below, and the control node is a computing device in the physical machine dimension), and the data node performs the global scheduling process of the tasks to be scheduled. This embodiment does not impose any restrictions on the selection of the control node, and specific settings can be made according to the actual application scenario; in one example, a data node can be randomly selected from the distributed system as a control node, and the control node performs the global scheduling process of the tasks to be scheduled; in another example, based on the total resources of each data node in the distributed system, the one with the largest total resources can be selected as the control node, and the control node performs the global scheduling process of the tasks to be scheduled; in another example, the control node is pre-set by the user or the system.

[0037] In some embodiments, each data node in the distributed system can periodically report node information of the data node to the control node, and the node information is relevant information about the data node, such as the node information includes but is not limited to the allocatable resources of the data node, the allocatable resources of each NUMA node in the data node and / or the load of each NUMA node, etc.

[0038] Exemplarily, for example, regarding the load of each NUMA node in the data node, the data node may determine the load of the NUMA node based on the statistical value of the utilization of multiple processors in the NUMA node, and the statistical value includes but is not limited to the average value, median value, maximum value or minimum value, etc. In one example, assuming that the NUMA node has multiple CPUs, after obtaining the CPU utilization of each of the multiple CPUs of the NUMA node, the data node may determine the load of the NUMA node based on the average value of all CPU utilizations, so as to report it to the control node.

[0039] When performing task scheduling, the control node can obtain the resources required for the task to be scheduled, and obtain the allocatable resources of each data node from the node information reported by each data node, and then determine at least one first target data node whose allocatable resources are greater than the required resources based on the resources required for the task to be scheduled and the allocatable resources of each data node. The first target data node is the data node whose quota meets the task to be scheduled, and then one or more target NUMA nodes suitable for the task to be scheduled can be found among the first target data nodes.

[0040] For each determined first target data node, the control node may determine one or more target NUMA nodes in the first target data node that are suitable for running the task to be scheduled based on the resources required by the task to be scheduled and the allocatable resources of each NUMA node.

[0041] In a possible implementation, in the process of determining one or more target NUMA nodes suitable for the task to be scheduled, for each first target data node, first determine whether the allocatable resources of the NUMA node with the lowest load meet the resources required by the task to be scheduled; if so, determine the NUMA node with the lowest load as the target NUMA node; if not, add other NUMA nodes in order from small to large according to the load of the NUMA node, until the sum of the allocatable resources of at least two added NUMA nodes meets the resources required by the task to be scheduled or all NUMA nodes have been added, and determine the at least two added NUMA nodes as the target NUMA nodes. In this embodiment, the NUMA node with the lowest load can be screened, which is conducive to quickly finding one or more target NUMA nodes suitable for the task to be scheduled, and when the allocatable resources of the NUMA node with the lowest load meet the resources required by the task to be scheduled, the task to be scheduled can be deployed on the NUMA node with the lowest load without being deployed on multiple NUMA nodes, which can further reduce the problems of frequent CPU switching and memory access across NUMA nodes, and is conducive to improving the performance of task deployment on data nodes.

[0042] Of course, other methods may also be used to determine one or more target NUMA nodes in each of the first target data nodes that are suitable for the execution of the task to be scheduled. The embodiments of this specification do not impose any restrictions on this. For example, one or more target NUMA nodes suitable for the execution of the task to be scheduled may be selected in order of load from high to low in each of the first target data nodes.

[0043] After determining one or more target NUMA nodes among each of the first target data nodes that are suitable for running the task to be scheduled, the control node can select the second target data node with a lower load from the first target data nodes based on the load of one or more target NUMA nodes among each of the first target data nodes, for example, the load of the second target data node is lower than at least half of the load of the first target data node, thereby ensuring that there are sufficient resources on the screened second target data node to run the task to be scheduled.

[0044] In a possible implementation, in the process of selecting the second target data node, for each of the first target data nodes, if the first target data node includes a target NUMA node, the control node may determine an evaluation score based on the load of the target NUMA node. Exemplarily, the evaluation score is equal to the load of the target node or the evaluation score is the load of the target node obtained by a preset operation (such as multiplying by a preset coefficient). Exemplarily, the load of the target NUMA node may be determined based on the statistical value of the utilization of multiple processors in the target NUMA node, such as the load of the target NUMA node is the average value of the utilization of all processors in the target NUMA node.

[0045] If the first target data node includes multiple target NUMA nodes, the evaluation score can be determined according to the statistical values ​​of the loads of the multiple target NUMA nodes. The statistical values ​​include but are not limited to the average, median, maximum value or minimum value, etc.; for example, the evaluation score can be determined according to the average value of the loads of the multiple target NUMA nodes, and the evaluation score is equal to the average value or the evaluation score is the average value of the loads of the target NUMA nodes obtained by a preset operation (such as multiplying by a preset coefficient).

[0046] The evaluation score reflects the load of one or more target NUMA nodes. After determining the evaluation scores of each first target data node, the first target data node with a relatively low evaluation score can be determined as the second target data node, thereby ensuring that there are sufficient resources on the determined second target data node to run the task to be scheduled. Optionally, in order to further improve the performance of the task to be scheduled on the data node, the first target data node with the lowest evaluation score (reflecting the lowest load) can be determined as the second target data node.

[0047] After determining the second target data node, the control node can deploy the task to be scheduled to one or more target NUMA nodes in the second target data node. This embodiment can screen out one or more target NUMA nodes that are suitable for running the task to be scheduled from a global perspective, improve the performance of deploying the task to be scheduled on the data node, and realize a more refined scheduling process.

[0048] In an example, see Figure 5, take two data nodes 100 as an example for explanation, the data node 100 includes multiple NUMA nodes, the NUMA node includes multiple processors with shared memory, and one or more tasks are running on the NUMA node; one of the data nodes is a control node, and the control node first determines the first target data node whose allocatable resources are greater than the resources required by the task to be scheduled from the two data nodes, assuming that the allocatable resources of the two data nodes both meet the resources required by the task to be scheduled, that is, the two data nodes are the first target data nodes. Next, determine one or more target NUMA nodes suitable for the task to be scheduled in each data node, and determine the evaluation score according to the load of the one or more target NUMA nodes suitable for the task to be scheduled. Assume that the target NUMA node suitable for the task to be scheduled in data node 1 is NUMA1, and the corresponding evaluation score is A; data node 2 is NUMA1+NUMA2, and the corresponding evaluation score is B; where A<B, then data node 1 can be determined as the second target data node, such as Figure 5 As shown, the control node can deploy the task to be scheduled to NUMA1 of data node 1.

[0049] In some embodiments, the node information regularly reported by each data node includes the allocatable resources of the data node. Generally, the allocatable resources of each data node are fixed, for example, the allocatable resources of the data node are equal to the total resources of the data node. However, considering that the actual load (or the resources actually used) of the task deployed on the NUMA node of the data node is dynamically changing, it is possible that the resources actually used by the task are less than the resources allocated to the task, which may cause a resource waste problem, or it is possible that the resources actually used by the task are greater than the resources allocated to the task. Therefore, in order to improve the resource utilization rate in the data node, moderate overselling can be performed, that is, the allocatable resources of the data node are dynamically adjusted according to the actual load situation (or the actual resource usage situation) and resource allocation situation of the data node, so that the data node can deploy more tasks when it is idle, thereby improving the overall resource utilization efficiency of the data node.

[0050] When each data node determines the allocatable resources of the data node, it first obtains the load of the data node. Exemplarily, the load of the data node can be determined based on the statistical values ​​of the loads of multiple NUMA nodes in the data node, and the load of the NUMA node can be determined based on the statistical values ​​of the utilization of multiple processors in the NUMA node; in other words, the load of the data node can be determined based on the statistical values ​​of the utilization of all processors in the data node. In one example, the load of the data node is: the average value of the loads of all NUMA nodes in the data node. Next, the data node can determine whether to oversell according to the load of the data node. For example, if the load of the data node is greater than a preset threshold, it means that the actual resources used by the task deployed on the NUMA node of the data node are relatively large. In order to ensure the normal operation of the deployed tasks and determine the stability of the data node, in this case, overselling is not performed or overselling is stopped. The allocatable resources of the data node are the first allocatable resources, which are equal to the total resources of the data node. If the load of the data node is less than or equal to the preset threshold, it means that the actual resources used by the task deployed on the NUMA node of the data node may be less than the resources allocated to the task, which may cause resource waste. In order to improve resource utilization, moderate overselling can be performed in this case, and the allocatable resources of the data node are dynamically adjusted to the second allocatable resources, which are greater than the first allocatable resources. In this embodiment, the allocatable resource information can be dynamically updated according to the load of the data node, and tasks that cannot be deployed to the data node are added to the data node with low load, so as to maximize the utilization of resources.

[0051] It can be understood that the preset threshold can be specifically set according to the actual application scenario, and this embodiment does not impose any restrictions on this. For example, the preset threshold is greater than 50%, such as 70% or 80%, etc.; for example, the preset threshold can be determined based on the allocated resources in the data node. For example, when the load of the data node is determined based on the load of the NUMA node (that is, the utilization of the processor), the preset threshold is the ratio of the allocated resources to the total resources.

[0052] The expected goal of task scheduling is usually to expect the resources allocated to the task to be close to the resources used by the task. Assuming that the resources allocated to the task are usage and the resources used by the task are request, then (1); Before overselling, the allocatable resources in the data node are generally equal to the total resources. Let the total resources of the data node be capacity and the allocatable resources be allocatable, then (2); Combining formula (1) and formula (2), we can get (3).

[0053] When the data node is oversold, that is, in the process of determining the second allocatable resource, according to formula (3), it can be determined that the size of the second allocatable resource is negatively correlated with the load of the data node. That is, when the load of the data node is relatively low, that is, when the tasks deployed in the NUMA node of the data node use less resources, the second allocatable resource increases, that is, the second allocatable resource can be set to be larger, so that more tasks can be deployed in the data node; when the load of the data node is relatively high, that is, when the tasks deployed in the NUMA node of the data node use more resources, the second allocatable resource decreases, that is, the second allocatable resource can be set to be smaller, so as to avoid the deployment of other tasks in the data node affecting the normal operation of the deployed tasks.

[0054] In some possible implementations, the second allocatable resource may be the minimum of the following three: ① the product of a preset oversold ratio and the total resources of the data node, the preset oversold ratio being greater than 1; the preset oversold ratio may be specifically set according to the actual application scenario, and this embodiment does not impose any restrictions on this. ② the ratio of the product of the total resources of the data node and the total allocated resources to the total used resources; this ratio takes into account all tasks running in the data node, and the ratio is converted based on the above formula (3). ③ The ratio of the product of the total resources of the data node and the allocated resources of the tasks of the specified priority to the resources used by the tasks of the specified priority; the ratio takes into account the tasks of the specified priority running in the data node, and the ratio is calculated based on the above formula (3); illustratively, there are tasks of the first priority and tasks of the second priority running in the data node, and the delay requirement of the tasks of the second priority is higher than the delay requirement of the tasks of the first priority, that is, the tasks of the second priority are delay-sensitive tasks that require the fastest possible response and have high real-time requirements, such as web page loading tasks, live broadcast tasks, etc.; the tasks of the first priority are best-effort tasks, such as data download tasks that do not require high real-time requirements; in an example, in order to meet the real-time requirements of the tasks of the second priority, the normal operation of the tasks of the second priority needs to be guaranteed, and the specified priority may be the second priority.

[0055] In one example, let the second allocatable resource be allocatable2, the preset oversold ratio be w, the total resource of the data node be capacity, the total allocated resource be total.usage, the total used resource be total.request, the allocated resource of the task of the specified priority be LS.usage, and the used resource of the task of the specified priority be LS.request, then In this embodiment, the second allocatable resource takes the minimum value from the above three, which can avoid the second allocatable resource being too large, resulting in excessive over-allocation of resources, and failure to provide sufficient resources when there is a burst of traffic (ie, the tasks deployed on the data node need to use more resources).

[0056] Of course, any one of the above three may be selected as the second allocatable resource of the data node according to actual application scenarios, and this embodiment does not impose any limitation on this.

[0057] In some possible implementations, an allocatable resource may be pre-set according to an actual application scenario. The preset allocatable resource is less than the total resources of the data node and greater than half of the total resources of the data node. For example, the preset allocatable resource is 80% of the total resources of the data node. When the data node determines the second allocatable resource, if the smallest of the three is less than the preset allocatable resource, the second allocatable resource is determined to be the preset allocatable resource. This embodiment sets a preset value to effectively prevent the second allocatable resource from being set too small, resulting in a situation where the data node has resources but cannot be allocated.

[0058] For example, if there are 100 CPUs in a data node, 1 task is deployed in the data node, 1 CPU is allocated, but 10 CPUs are actually used, then based on the above calculation method, allocatable2=10 (100*1 / 10=10), but there are actually 90 CPUs available. In this case, the second allocable resource is set too small. Therefore, a preset allocable resource can be used to prevent this situation from happening, such as setting the preset allocable resource to 80% of the total resources of the data node, that is, allocatable2=80, which is closer to the actual situation. In this embodiment, by setting the second allocable resource to the minimum of the above three, it is helpful to avoid the second allocable resource being set too large and causing over-allocation of resources. When the minimum of the above three is still less than the preset allocable resource, the second allocable resource is determined as the preset allocable resource, which is helpful to avoid the second allocable resource being set too small, resulting in the data node having resources but unable to allocate them, and ensuring that the second allocable resource is within a reasonable and effective range.

[0059] For each data node, after determining the new allocatable resources, the data node can update the allocatable resource information in the node information reported regularly to achieve the overselling effect. In one example, for example, please refer to 5, for NUMA1 of data node 1, it has deployed 3 tasks and allocated all resources, but the actual resources used by the 3 tasks are less than the allocated resources. Then, through the above-mentioned overselling strategy, the allocatable resources of the data node are dynamically adjusted, and the tasks to be scheduled can still be deployed to NUMA1 of data node 1, so as to increase the tasks that cannot be deployed to the data node with low load, so as to maximize the utilization of resources and achieve the overselling effect of the data node.

[0060] It should be noted that in the embodiments of this specification, it is the allocatable resources of the data nodes that are oversold, while the allocatable resources of the NUMA nodes in the data nodes are not oversold.

[0061] In some embodiments, in each data node, the data node may be oversold during the task running process, resulting in a local NUMA hotspot, that is, the resources actually used by the task deployed on the NUMA are greater than the total resources of the NUMA node. In order to eliminate the hotspot and ensure the smooth running of tasks with high real-time requirements, the data node can detect that the load of the NUMA node exceeds the first preset threshold, and freeze the tasks of the first priority running in the NUMA node; wherein the NUMA node runs tasks of the first priority and tasks of the second priority, the delay requirement of the tasks of the second priority is higher than the delay requirement of the tasks of the first priority, and the tasks of the second priority are delay-sensitive tasks that require the fastest response possible and have high real-time requirements, such as web page loading tasks, live broadcast tasks, etc.; the tasks of the first priority are best-effort tasks, such as data download tasks with low real-time requirements, etc.; this embodiment ensures the smooth execution of tasks with high real-time requirements by freezing tasks with low real-time requirements to release resources.

[0062] Exemplarily, in the process of freezing tasks of the first priority, the data node determines the usage resources of all the tasks of the first priority running on the NUMA node, and then freezes the tasks of the first priority in order of the usage resources from high to low, until the load of the NUMA node does not exceed the first preset threshold. In this embodiment, freezing in order of the usage resources from high to low is conducive to the rapid release of resources and further ensures the smooth execution of tasks with higher real-time performance. Of course, other orders can also be used to freeze the tasks of the first priority, and this embodiment does not impose any restrictions on this.

[0063] In an example, see Figure 5 , the NUMA1 node load in data node 2 is high, for example, exceeding the first preset value, then data node 2 can freeze the tasks (task 1 and task 3) belonging to the first priority, Figure 2 Medium grey indicates a frozen state, ensuring the smooth execution of tasks belonging to the second priority.

[0064] In some embodiments, if the load of the NUMA node still exceeds the first preset threshold after freezing all the tasks of the first priority, in order to ensure the smooth operation of the tasks with higher real-time requirements, the data node can migrate the tasks of the second priority in the NUMA node to other NUMA nodes; the load of the other NUMA nodes is lower than the load of the current NUMA node. This embodiment adopts a strategy of combining the freezing of tasks of the first priority and the migration of tasks of the second priority, which effectively solves the NUMA hot spot problem when there may be high load, and ensures the smooth operation of tasks with high real-time requirements (tasks of the second priority).

[0065] Exemplarily, in the process of migrating tasks belonging to the second priority, the data node can determine the usage resources of all the tasks belonging to the second priority running on the NUMA node, and then migrate the tasks belonging to the second priority to other NUMA nodes in order of the usage resources from low to high, until the load of the NUMA node does not exceed the first preset threshold or the load of the other NUMA nodes is higher than the load of this NUMA node. In this embodiment, it is considered that the greater the usage resources (such as CPU or memory usage), the greater the impact during the migration process. In order to reduce this impact, this specification implements the migration process of tasks belonging to the second priority in order of the usage resources from low to high, reducing the fluctuations generated during the migration process. Of course, other orders can also be used to migrate the tasks belonging to the second priority, and this embodiment does not impose any restrictions on this.

[0066] In some embodiments, if the load of the NUMA node of the data node is lower than a second preset threshold, the data node may wake up the frozen task of the first priority; wherein the second preset threshold is less than the first preset threshold, so as to fully utilize the resources of the data node. In one example, for example, the first preset threshold is 80% of the overall utilization of the data node, and the second preset threshold is 60% of the overall utilization of the data node.

[0067] In an exemplary application scenario, the distributed system is taken as an example of K8s cluster. Kubernetes, referred to as K8s, is an abbreviation formed by replacing the 8 characters "ubernete" with 8. It is a distributed container cluster management platform that provides a series of complete functions such as deployment and operation, resource scheduling, service discovery and dynamic scaling for containerized applications. Kubernetes can be deployed in the form of a cluster. The data nodes in the Kubernetes cluster may include but are not limited to the master node (Master) and multiple worker nodes (node), which can be physical machines (such as physical servers) or virtual machines. Applications can be deployed on worker nodes, and applications can be deployed on certain specific worker nodes through configuration.

[0068] In Kubernetes, Pod is the basic scheduling unit of the Kubernetes cluster, which contains a group of containers and data volumes. Pod logically represents an instance of a certain application. For example, a web site application is built by the front-end, back-end and database. These three components will run in their own containers, so a pod containing three containers can be created.

[0069] For example, see Figure 6 The data nodes in the K8s cluster include a master node and multiple worker nodes. The worker nodes include multiple NUMA nodes 10. Each NUMA node 10 includes multiple processors with shared memory. The NUMA nodes can run one or more tasks. In order to implement the task scheduling process mentioned in this specification, a task scheduling module 20 can be deployed in the master node, and the task scheduling module can be executed by the processor in the master node; an evaluation module 21, an oversold module 22 and an adjustment module 23 are deployed in each worker node, and the evaluation module, oversold module and adjustment module can be executed by the processor in the worker node.

[0070] The evaluation module 21 is configured to collect the utilization of each processor in the NUMA node, and determine the load information of the NUMA node according to the average utilization of all processors in the NUMA node. For example, if the load information of the NUMA node is NUMA score, the utilization of the processor is CPUusage, and the NUMA node has i processors, i is an integer greater than 0, then .

[0071] The overselling module 22 is configured to: dynamically determine the allocatable resources of the working node according to the load of the working node, for example, if the load of the working node is greater than a preset threshold, the allocatable resources of the working node are first allocatable resources, and the first allocatable resources are equal to the total resources of the working node; if the load of the working node is less than or equal to the preset threshold, the allocatable resources of the working node are second allocatable resources, and the second allocatable resources are greater than the first allocatable resources.

[0072] Exemplarily, the second allocatable resource is the minimum of the following three: ① the product of a preset oversold ratio and the total resources of the working node, wherein the preset oversold ratio is greater than 1; ② the ratio of the product of the total resources of the working node and the total allocated resources to the total used resources; ③ the ratio of the total resources of the working node and the allocated resources of the tasks of the specified priority to the used resources of the tasks of the specified priority.

[0073] Exemplarily, a preset allocatable resource is defined, and the preset allocatable resource is less than the total resources of the working node and greater than half of the total resources of the working node; if the smallest of the three is less than the preset allocatable resource, the second allocatable resource is the preset allocatable resource. In this embodiment, by setting the second allocatable resource to be the smallest of the above three, it is helpful to avoid the second allocatable resource being set too large, resulting in over-allocation of resources. When the smallest of the above three is still less than the preset allocatable resource, the second allocatable resource is determined as the preset allocatable resource, which is helpful to avoid the second allocatable resource being set too small, resulting in the working node having resources but being unable to allocate them, and ensuring that the second allocatable resource is within a reasonable and effective range.

[0074] The overselling module 22 can dynamically oversell the overall resources at the work node granularity, and can deploy more tasks on the work node when the load is low, thereby alleviating the problem of insufficient overall resource utilization and effectively improving the resource utilization efficiency of the work node.

[0075] After determining the load of each NUMA node in the working node and the new allocatable resources, the working node can report node information including NUMA load information, allocatable resources of the NUMA node, allocatable resources of the working node, etc., wherein the allocatable resources of the NUMA node are generally fixed, and the allocatable resources of the working node change dynamically according to the load of the working node. Since the load of each NUMA node in the working node and the new allocatable resources change dynamically, the working node can report node information regularly.

[0076] The task scheduling module 20 is configured to: receive the node information reported by each working node; when making a scheduling decision, first determine at least one first target working node whose allocatable resources are greater than the required resources based on the resources required for the task to be scheduled and the allocatable resources of each working node; for each first target working node, list each NUMA node in order from small to large according to the load, and if the allocatable resources of the first NUMA node meet the required resources of the task to be scheduled, then directly return the NUMA score of the NUMA node as the evaluation score. If not, add the second NUMA node to the result set, determine whether the sum of the allocatable resources of the two NUMA nodes meets the required resources of the task to be scheduled, if so, return the average of the two NUMA scores as the evaluation score, if not, add the third NUMA node to the result set, and so on, until the sum of the allocatable resources of all added NUMA nodes meets the required resources of the task to be scheduled or all NUMA nodes have been added to the result set. After determining the evaluation scores of each of the first target work nodes, the first target work node with the smallest evaluation score is determined as the second target work node, and the task to be scheduled can be deployed to one or more target NUMA nodes in the second target work node.

[0077] The task scheduling module 20 can select the most suitable NUMA node for scheduling among the numerous working nodes and NUMA in the cluster, solving the problem of refined scheduling on large computing machines, reducing frequent CPU switching, CPU current limiting, cross-NUMA memory access, etc., and improving task processing performance.

[0078] The adjustment module 23 is configured to perform the following process:

[0079] (1) When the load of the NUMA node of the working node exceeds the first preset threshold, the freezing process of the tasks of the first priority is triggered, and all the tasks of the first priority deployed on the NUMA are listed, sorted from high to low according to the resources used, and the freezer.state value of the tasks of the first priority is set to FROZEN one by one through the freeze module in cgroups to release the resources of the task. After freezing each task, check whether the load of the NUMA node is reduced below the first preset threshold. If it fails to be reduced below the first preset threshold, continue to execute this step until the load of the NUMA node is reduced below the threshold or the tasks of the first priority can be frozen.

[0080] (2) If after freezing all tasks of the first priority, the load of the NUMA node still exceeds the first preset threshold, list all tasks of the second priority that are deployed separately on the NUMA, sort them from high to low according to the resources used, and modify the CPUSet of the second priority from low to high in sequence, migrating the second priority tasks from the NUMA node with a higher load to the NUMA node with a lower load, until the NUMA load drops below the first preset threshold or the loads of the NUMA nodes are relatively balanced.

[0081] (3) If the load of the NUMA node of the working node is lower than the second preset threshold, and the second preset threshold is lower than the first preset threshold, the freezer.state value of the tasks belonging to the first priority is set to THAWED one by one through the freeze module in cgroups, and the tasks belonging to the first priority are woken up one by one to make full use of the overall resource utilization.

[0082] The adjustment module 23 can use the characteristics of cgroup to freeze the tasks of the first priority when hot spots appear in NUMA. In extreme cases, it can migrate the tasks of the second priority from the NUMA with higher load to the NUMA with lower load, thereby ensuring the smooth operation of the tasks of the second priority.

[0083] The modules listed above are only illustrative, and the modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this specification. Ordinary technicians in this field can understand and implement it without creative work.

[0084] Accordingly, see Figure 3 , the present description embodiment also provides a distributed system, the distributed system includes a plurality of data nodes, the data nodes include one or more NUMA nodes;

[0085] The data node is used to execute any one of the above methods.

[0086] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of the device to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0087] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a terminal, enables the terminal to execute the above method.

[0088] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0089] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0090] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0091] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0092] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0093] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0094] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0095] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0096] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A task scheduling method, applied to a distributed system, wherein the distributed system includes a plurality of data nodes, and the data nodes include a plurality of NUMA nodes; the method comprises: According to the resources required by the task to be scheduled and the allocatable resources of each of the data nodes, at least one first target data node whose allocatable resources are greater than the required resources is determined; wherein, if the load of the data node is greater than a preset threshold, the allocatable resources of the data node are first allocatable resources, and the first allocatable resources are equal to the total resources of the data node; if the load of the data node is less than or equal to the preset threshold, the allocatable resources of the data node are second allocatable resources, and the second allocatable resources are greater than the first allocatable resources; For each of the first target data nodes, determining one or more target NUMA nodes suitable for running the task to be scheduled according to the resources required by the task to be scheduled and the allocatable resources of each of the NUMA nodes; Selecting a second target data node from the first target data nodes according to the load of one or more target NUMA nodes among the first target data nodes; the load of the second target data node is lower than at least half of the load of the first target data node; Deploy the task to be scheduled to one or more target NUMA nodes in the second target data node.

2. The method according to claim 1, wherein determining one or more target NUMA nodes suitable for running the task to be scheduled according to the resources required by the task to be scheduled and the allocatable resources of each of the NUMA nodes comprises: Determine whether the allocatable resources of the NUMA node with the lowest load meet the resources required by the task to be scheduled; If so, determining the NUMA node with the lowest load as the target NUMA node; If not, other NUMA nodes are added in order of the NUMA node load from small to large until the sum of the allocatable resources of at least two added NUMA nodes meets the resources required by the task to be scheduled or all NUMA nodes have been added, and the at least two added NUMA nodes are determined as the target NUMA nodes.

3. The method according to claim 1 or 2, wherein selecting a second target data node from the first target data node according to the load of one or more target NUMA nodes in the first target data node comprises: If the first target data node includes a target NUMA node, determining an evaluation score according to a load of the target NUMA node; If the first target data node includes multiple target NUMA nodes, determining the evaluation scores according to the statistical values ​​of the loads of the multiple target NUMA nodes; Determine the first target data node with the lowest evaluation score as the second target data node; The NUMA node includes multiple processors; the load of the NUMA node is determined according to the statistical value of the utilization of the multiple processors in the NUMA node. 4 . The method according to claim 1 , wherein the load of the data node is determined based on statistical values ​​of loads of multiple NUMA nodes in the data node. 5 . The method according to claim 4 , wherein the size of the second allocatable resource is negatively correlated with the load of the data node.

6. The method according to claim 4, wherein the second allocatable resource is the smallest of the following three: a product of a preset oversold ratio and total resources of the data node, wherein the preset oversold ratio is greater than 1; The ratio of the product of the total resources and the total allocated resources of the data node to the total used resources; The ratio of the product of the total resources of the data node and the allocated resources of the task of the specified priority to the used resources of the task of the specified priority.

7. According to the method of claim 6, if the smallest of the three is less than the preset allocatable resources, then the second allocatable resources are the preset allocatable resources, and the preset allocatable resources are less than the total resources of the data node and greater than half of the total resources of the data node.

8. The method according to claim 1, further comprising: For each of the data nodes, if the load of the NUMA node in the data node exceeds a first preset threshold, the tasks of the first priority running in the NUMA node are frozen; wherein the NUMA node runs tasks of the first priority and tasks of the second priority, and the latency requirement of the tasks of the second priority is higher than the latency requirement of the tasks of the first priority.

9. The method according to claim 8, wherein freezing the tasks of the first priority level running in the NUMA node comprises: Determine the usage resources of all tasks belonging to the first priority running on the NUMA node; The tasks of the first priority are frozen in order from high to low resource usage until the load of the NUMA node does not exceed the first preset threshold.

10. The method according to claim 8, further comprising: If the load of the NUMA node of the data node is lower than a second preset threshold, the frozen task of the first priority is awakened; wherein the second preset threshold is smaller than the first preset threshold.

11. The method according to claim 8, further comprising: If the load of the NUMA node still exceeds the first preset threshold after all tasks of the first priority are frozen, the tasks of the second priority in the NUMA node are migrated to other NUMA nodes; the load of the other NUMA nodes is lower than the load of the current NUMA node.

12. The method according to claim 11, wherein migrating the tasks of the second priority in the NUMA node to other NUMA nodes comprises: Determine the usage resources of all the tasks belonging to the second priority running on the NUMA node; The tasks of the second priority are migrated to other NUMA nodes in order of resource usage from low to high, until the load of the NUMA node does not exceed the first preset threshold or the load of the other NUMA nodes is higher than the load of the current NUMA node.

13. A distributed system, comprising a plurality of data nodes, wherein the data nodes comprise one or more NUMA nodes; The data node is used to execute the method according to any one of claims 1 to 12.

14. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

Images