Resource scheduling method and device, electronic equipment and storage medium

Abstract

This invention provides a resource scheduling method, apparatus, electronic device, and storage medium. The method includes: receiving tasks to be processed and submitting them to a cluster resource manager; determining the resource fragmentation rate in the cluster resources; determining a resource scheduling mode matching the cluster resources based on the resource fragmentation rate, wherein the resource scheduling mode includes: a forced high-low affinity mode and a mild high-low affinity mode; and configuring corresponding cluster resources for the tasks to be processed according to their priority. This allows for the flexible determination of a resource scheduling mode matching the cluster resources based on the resource fragmentation rate. The forced high-low affinity mode and the mild high-low affinity mode can reduce the resource fragmentation rate in the cluster resources, improving resource utilization efficiency, while ensuring timely processing of high-priority tasks, guaranteeing data processing speed for users, and improving the user experience.

Description

Technical Field

[0001] This invention relates to resource scheduling and processing technology for cloud networks, and more particularly to resource scheduling methods, devices, electronic equipment, and storage media. Background Technology

[0002] With the continuous development of computer technology, cloud servers (CVMs) can provide secure and reliable elastic computing services, and offer different instance types to meet specific user scenarios. These instance types consist of different combinations of CPU, memory, storage, and network. When cloud servers perform resource scheduling, the speed of resource scheduling directly affects the resource utilization and user experience of the cloud data center. Scheduling prioritizes ensuring that users can be allocated resources, followed by how to optimally allocate resources, i.e., improve resource utilization. Related scheduling algorithms, including the binpack algorithm, have the drawback of not considering scenarios with multiple priority tasks. Using it in such scenarios can lead to resource fragmentation for higher-priority tasks. Rescheduling algorithms have the drawback of needing to kill and restart already running tasks, which affects tasks, and cannot be used for tasks that cannot be restarted. Virtualization / pooling technology has the drawback of affecting task execution speed. Predicting task completion time often suffers from low accuracy, making it impractical for real-world applications. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a resource scheduling method, apparatus, electronic device, and storage medium that can flexibly determine a resource scheduling mode that matches the cluster resources based on the resource fragmentation rate in the cluster resources. By using a forced high-low affinity mode and a mild high-low affinity mode, the resource fragmentation rate in the cluster resources can be reduced, thereby improving the utilization efficiency of resources. At the same time, it can ensure the timely processing of high-priority tasks, guarantee the data processing speed of cloud server users, and improve the user experience.

[0004] The technical solution of this invention is implemented as follows: This invention provides a resource scheduling method, the method comprising: Receive the tasks to be processed and submit them to the cluster resource manager; The cluster resource manager is used to determine the resource fragmentation rate in the cluster resources; Based on the resource fragmentation rate in the cluster resources, a resource scheduling mode matching the cluster resources is determined, wherein the resource scheduling mode includes: a forced high-low affinity mode and a mild high-low affinity mode. In response to a resource scheduling mode that matches the cluster resources, corresponding cluster resources are configured for the tasks to be processed according to their priority.

[0005] This invention also provides a resource scheduling device, comprising: The information transmission module is used to receive tasks to be processed and submit the tasks to be processed to the cluster resource manager; The information processing module is used to determine the resource fragmentation rate in the cluster resources through the cluster resource manager; The information processing module is used to determine a resource scheduling mode that matches the cluster resources based on the resource fragmentation rate in the cluster resources. The resource scheduling mode includes: a forced high-low affinity mode and a mild high-low affinity mode. The information processing module is used to configure corresponding cluster resources for the tasks to be processed according to the priority of the tasks to be processed, in response to a resource scheduling mode that matches the cluster resources.

[0006] This invention also provides an electronic device, the electronic device comprising: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements a preceding resource scheduling method.

[0007] This invention also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement a preceding resource scheduling method.

[0008] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the resource scheduling method provided in this application.

[0009] The embodiments of the present invention have the following beneficial effects: This invention receives tasks to be processed and submits them to a cluster resource manager. The cluster resource manager then determines the resource fragmentation rate within the cluster resources. Based on this fragmentation rate, a resource scheduling mode matching the cluster resources is determined. This resource scheduling mode includes a forced high-low affinity mode and a moderate high-low affinity mode. In response to the resource scheduling mode matching the cluster resources, corresponding cluster resources are configured for the tasks to be processed according to their priority. This allows for flexible determination of a resource scheduling mode matching the cluster resources based on their fragmentation rate. The forced high-low affinity mode and the moderate high-low affinity mode reduce resource fragmentation, improving resource utilization efficiency, while ensuring timely processing of high-priority tasks, guaranteeing data processing speed for cloud server users, and improving the user experience. Attached Figure Description

[0010] Figure 1 This is a schematic diagram illustrating a use case of the resource scheduling method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the composition structure of the resource scheduling device provided in an embodiment of the present invention; Figure 3 This is an optional flowchart illustrating the resource scheduling method provided in an embodiment of the present invention. Figure 4 This is an optional flowchart illustrating the resource scheduling method provided in an embodiment of the present invention. Figure 5 This is a schematic diagram illustrating the testing process of a resource scheduling method in an embodiment of the present invention; Figure 6 This is an optional flowchart illustrating the resource scheduling method provided in an embodiment of the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0013] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be interpreted as follows.

[0014] 1) In response to, used to indicate the conditions or states on which the operation performed depends. When the conditions or states on which it depends are met, one or more operations performed may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0015] 2) Terminals, including but not limited to: ordinary terminals and dedicated terminals, wherein the ordinary terminals maintain a long connection and / or a short connection with the transmission channel, and the dedicated terminals maintain a long connection with the transmission channel.

[0016] 3) Client: The carrier that implements specific functions in the terminal. For example, a mobile client (APP) is a carrier of specific functions in a mobile terminal, such as performing report creation or report display functions.

[0017] 4) Components are functional modules of the view in a mini-program, also known as front-end components. These include buttons, titles, tables, sidebars, content, and footers on the page. Components include modular code so that they can be reused in different pages of the mini-program.

[0018] 5) Server cluster refers to a group of servers working together to provide the same service, appearing to the client as a single server. Server clusters can utilize multiple computers for parallel computing to achieve high computing speeds, and can also use multiple computers for backup, ensuring the entire system continues to operate normally even if one machine fails. The resource scheduling method provided in this application can be applied to cloud server and distributed server scenarios to achieve status detection and fault repair of server hard drives in different scenarios. Specifically, a cloud server (CVM Cloud Virtual Machine) is a simple, efficient, secure, reliable, and elastically scalable computing service. Its management is simpler and more efficient than traditional single physical servers. Users can quickly create or release any number of cloud servers for their business processes and store user data without purchasing hardware in advance. In a distributed server environment, user data and programs may not reside on a single server but are distributed across multiple servers. Similarly, distributed server environments also require a large number of hard drives, and the resource scheduling method provided in this application is needed to achieve server hard drive status detection and fault repair.

[0019] Figure 1 This is a schematic diagram illustrating a use case of the resource scheduling method provided in an embodiment of the present invention. See also: Figure 1 With the continuous development of computer technology, cloud servers (Cloud Virtual Machines, CVMs) can provide secure and reliable elastic computing services, and can also provide different instance types to meet specific user scenarios. Terminals (including terminals 10-1 and 10-2) are equipped with corresponding clients capable of performing different functions. These clients (including terminals 10-1 and 10-2) obtain different information from the corresponding cloud server 200 via network 300 and can deploy different services on the cloud server. The terminals connect to the cloud server 200 via network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both, using a wireless link for data transmission. The instance types provided by the cloud server consist of different combinations of CPU, memory, storage, and network, and store user business data on the cloud server's hard drive. However, during the operation of the cloud server, a large number of resource fragments are generated during task processing, causing resource redundancy, reducing the processing speed of the cloud server network, affecting task processing speed, and impacting the overall performance of the cloud server network. In the embodiments provided by this invention, the cloud server application running in the cloud server 200 can be written in software code environments of different programming languages, and the code objects can be different types of code entities. For example, in C language software code, a code object can be a function. In JAVA language software code, a code object can be a class, and in iOS Objective-C, it can be a piece of object code. In C++ language software code, a code object can be a class or a function to execute processing instructions from different terminals. This application does not distinguish the source of the cloud server's compilation environment.

[0020] The structure of the resource scheduling device according to an embodiment of the present invention will be described in detail below. The resource scheduling device can be implemented in various forms, such as a dedicated terminal with resource scheduling device processing function, or a server equipped with resource scheduling device processing function, for example, the preceding... Figure 1 The cloud server in the middle is 200. Figure 2 This is a schematic diagram of the composition structure of the resource scheduling device provided in the embodiment of the present invention. It can be understood that... Figure 2 The diagram shows only an exemplary structure of the resource scheduling device, not the entire structure; implementation is possible as needed. Figure 2 The structure shown may be part or all of the structure.

[0021] The resource scheduling device provided in this embodiment of the invention includes at least one processor 201, a memory 202, a user interface 203, and at least one network interface 204. The various components in the resource scheduling device are coupled together via a bus system 205. It can be understood that the bus system 205 is used to implement communication between these components. In addition to a data bus, the bus system 205 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 2 The general labeled all buses as Bus System 205.

[0022] The user interface 203 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0023] It is understood that memory 202 can be volatile memory or non-volatile memory, or both. In this embodiment of the invention, memory 202 is capable of storing data to support the operation of a terminal (such as 10-1). Examples of this data include any computer programs used to operate on the terminal (such as 10-1), such as operating systems and applications. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications.

[0024] In some embodiments, the resource scheduling device provided in this invention can be implemented using a combination of hardware and software. For example, the resource scheduling device provided in this invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the resource scheduling method provided in this invention. For instance, the processor in the form of a hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0025] As an example of the resource scheduling device provided in this embodiment of the invention, which is implemented by combining software and hardware, the resource scheduling device provided in this embodiment of the invention can be directly embodied as a combination of software modules executed by processor 201. The software modules can be located in a storage medium, which is located in memory 202. Processor 201 reads the executable instructions included in the software modules in memory 202 and combines them with necessary hardware (e.g., including processor 201 and other components connected to bus system 205) to complete the resource scheduling method provided in this embodiment of the invention.

[0026] As an example, processor 201 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0027] As an example of the hardware implementation of the resource scheduling device provided in the embodiments of the present invention, the device provided in the embodiments of the present invention can be directly executed by a processor 201 in the form of a hardware decoding processor. For example, it can be executed by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the resource scheduling method provided in the embodiments of the present invention.

[0028] In this embodiment of the invention, the memory 202 is used to store various types of data to support the operation of the resource scheduling device. Examples of such data include: any executable instructions for operation on the resource scheduling device, such as executable instructions, in which a program implementing the resource scheduling method of this embodiment of the invention may be included.

[0029] In other embodiments, the resource scheduling device provided in this invention can be implemented in software. Figure 2A resource scheduling device stored in memory 202 is shown. This device can be software in the form of programs and plug-ins, and includes a series of modules. As an example of a program stored in memory 202, it may include a resource scheduling device. The resource scheduling device includes the following software modules: an information transmission module 2081 and an information processing module 2082. When the software modules in the resource scheduling device are read into RAM and executed by processor 201, the resource scheduling method provided in this embodiment of the invention will be implemented. The functions of each software module in the resource scheduling device include: The information transmission module 2081 is used to receive tasks to be processed and submit the tasks to be processed to the cluster resource manager.

[0030] The information processing module 2082 is used to determine the resource fragmentation rate in the cluster resources through the cluster resource manager.

[0031] The information processing module 2082 is used to determine a resource scheduling mode that matches the cluster resources based on the resource fragmentation rate in the cluster resources. The resource scheduling mode includes: a forced high-low affinity mode and a mild high-low affinity mode.

[0032] The information processing module 2082 is used to configure corresponding cluster resources for the tasks to be processed according to the priority of the tasks to be processed, in response to a resource scheduling mode that matches the cluster resources.

[0033] according to Figure 2 The illustrated electronic device, in one aspect of this application, also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the resource scheduling method described above.

[0034] To overcome the drawback of excessive resource fragmentation caused by cloud server network usage, which affects task processing speed, refer to... Figure 3 , Figure 3 This is an optional flowchart illustrating the resource scheduling method provided in an embodiment of the present invention. It can be understood that... Figure 3 The steps shown can be performed by various electronic devices running resource scheduling devices, such as dedicated terminals with resource scheduling functions, servers or server cluster controllers, and control terminals of cloud network servers. The dedicated terminal with a resource scheduling device can be encapsulated within... Figure 1 In the cloud server 200 shown, to execute the preprocessor. Figure 2The corresponding software modules in the resource scheduling device shown. The following section addresses... Figure 3 The steps shown are explained.

[0035] Step 301: The resource scheduling device receives the task to be processed and submits the task to be processed to the cluster resource manager.

[0036] In this invention, embodiments can be implemented using cloud technology. Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. It can also be understood as a general term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on cloud computing business models. The backend services of network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites; therefore, cloud technology needs cloud computing as its support.

[0037] It's important to note that cloud computing is a computing model that distributes computing tasks across a resource pool comprised of numerous computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, resources in the "cloud" appear infinitely scalable, readily available, on-demand, and expandable, with payment based on usage. As a provider of fundamental cloud computing capabilities, a cloud resource pool platform, often referred to as a cloud platform or Infrastructure as a Service (IaaS), is established. This platform deploys various types of virtual resources within the resource pool for external customers to choose from. The cloud resource pool primarily includes: computing devices (which can be virtualized machines containing operating systems), storage devices, and network devices. When users use cloud servers to store data or deploy different application processes, monitoring the operating parameters of the server cluster hard drives allows for timely detection of potential server cluster hard drive failures, preventing data loss due to server cluster hard drive failures that trigger failure warnings.

[0038] Taking the resource scheduling method provided in this application implemented through a cloud server network as an example, tasks to be processed are received through different service cluster front-ends of the cloud computing server; system access components in the different service clusters are triggered, and the tasks to be processed are submitted to the cluster resource manager matching the different service clusters through the system access components in the different service clusters. The front-end can be a Web UI component used to receive Spark-related parameters filled in by the user and generate job data based on these parameters. The cluster manager can be an open-source resource scheduling platform such as YARN, Mesos, or Kubernetes. Spark itself already supports these open-source platforms, meaning the protocols between the Spark component and the ClusterManager component are compatible. Driver is the job driver, Work Node is the worker node, Executor is the task execution component, and task is the smallest unit of execution. Furthermore, the structured data package (Spark SQL) is the package used by Spark to manipulate structured data. Through Spark SQL, data can be queried using the SQL language. Spark SQL supports multiple data sources, such as data warehouse tools (Hive) tables. This streaming computing component is provided by Spark for performing streaming computing on real-time data, and provides an Application Programming Interface (API) for manipulating data streams.

[0039] Step 302: The resource scheduling device determines the resource fragmentation rate in the cluster resources through the cluster resource manager.

[0040] In some embodiments of the present invention, determining a resource scheduling mode that matches the cluster resources based on the resource fragmentation rate in the cluster resources can be achieved in the following ways: A resource fragmentation rate threshold matching the cluster resource manager is determined. When the resource fragmentation rate in the cluster resources is greater than the resource fragmentation rate threshold, the resource scheduling mode matching the cluster resources is determined to be a mild high-low affinity mode. When the resource fragmentation rate in the cluster resources is less than or equal to the resource fragmentation rate threshold, the resource scheduling mode matching the cluster resources is determined to be a forced high-low affinity mode. It should be noted that, due to the complex task processing environment faced by the cluster resource manager, for resource scheduling environments with many concurrent tasks, dynamic adjustment is not required, and only the forced high-low affinity mode can be used for resource scheduling. Similarly, for resource scheduling environments with low system resource consumption, dynamic adjustment is not required, and only the mild high-low affinity mode can be used for resource scheduling, thereby reducing the load on the corresponding cluster resource manager during resource scheduling.

[0041] Determine the resource fragmentation rate and corresponding threshold that match the cluster resource manager; the resource fragmentation rate can be derived from Formula 1, which can also be adjusted according to the needs of the scenario: Formula 1 In this application, the cloud computing platform uses machines with 8 resources. In Formula 1, the number of idle high-priority machines refers to the number of machines with 8 resources that are idle or used by low-priority tasks. The number of idle high-priority resources is the sum of the number of low-priority task resources and the number of idle resources. In practice, the resource fragmentation rate threshold can be set to 0.8 based on experience. Other thresholds can be selected according to the characteristics of the usage scenario.

[0042] In some embodiments of the present invention, a monitoring process matching the cluster resource manager can be triggered; the resource fragmentation rate in the cluster resources is monitored in real time through the monitoring process matching the cluster resource manager; when the resource fragmentation rate in the cluster resources reaches the resource fragmentation rate threshold, the resource scheduling mode matching the cluster resources is dynamically adjusted. Since the number of cluster resources in the cloud server network changes, possibly due to continuous addition of new cloud servers by operators, and the number of tasks received by the cloud server network also changes, real-time monitoring of fragmentation levels allows for flexible adjustment of the resource scheduling mode, providing users of cloud network products with a better user experience.

[0043] Step 303: The resource scheduling device determines a resource scheduling mode that matches the cluster resources based on the resource fragmentation rate in the cluster resources.

[0044] The resource scheduling modes include: a forced high-low affinity mode and a mild high-low affinity mode.

[0045] Step 304: The resource scheduling device responds to the resource scheduling mode that matches the cluster resources and configures the corresponding cluster resources for the tasks to be processed according to their priority.

[0046] In some embodiments of the present invention, when the resource scheduling mode is a mild high-low affinity mode, the total amount of node resources in the cluster resources is determined; a resource scheduling subprocess in the mild high-low affinity mode is triggered, and the resource scheduling subprocess configures corresponding cluster resources for the tasks to be processed according to their priority; or a resource preemption subprocess in the mild high-low affinity mode is triggered, and the resource preemption subprocess configures corresponding cluster resources for the tasks to be processed according to their priority. The following describes... Figure 4 To further explain, the working process of the mild, high-low affinity mode.

[0047] refer to Figure 4 , Figure 4 This is an optional flowchart illustrating the resource scheduling method provided in an embodiment of the present invention. It can be understood that... Figure 4 The steps shown can be performed by various electronic devices running resource scheduling devices, such as dedicated terminals with resource scheduling functions, servers or server cluster controllers, and control terminals of cloud network servers. The dedicated terminal with a resource scheduling device can be encapsulated within... Figure 1 In the cloud server 200 shown, to execute the preprocessor. Figure 2 The corresponding software modules in the resource scheduling device shown. The following section addresses... Figure 4 The steps shown are explained.

[0048] Step 401: When configuring corresponding cluster resources for the tasks to be processed according to their priority, the resource scheduling subprocess determines whether the idle node resources in the cluster resources meet the requirements of the tasks to be processed.

[0049] Step 402: When the idle node resources in the cluster resources meet the task to be processed, the task to be processed is processed through the idle node resources.

[0050] The resource scheduling subprocess processing algorithm can be represented by Equation 2: Formula 2 Where C represents the number of resources, C_same priority represents the number of resources with the same priority on the node, and so on.

[0051] The resource scheduling subprocess's processing algorithm can be represented by the following pseudocode: Func calculateScoreSoftAllocate(node, task){ score = node.capacity / / Initialize the score as the total node resources. The goal is to ensure the score is greater than or equal to 0. if task.requiredResource>node.idle return 0 / / If available resources are insufficient to meet the task requirements, no points are awarded. if task.priority == highPriority { / / If it's a high-priority task score += node.highUsageResource W_2 - node.lowUsageResource W_3 / / The more resources are occupied by the same priority, the higher the score. highUsageResource represents the amount of resources occupied by high priority. Assume there are two tasks on the node: high priority A requires 1 resource and low priority B requires 2 resources, and the total resources of the node are 8. For the high priority task to be scheduled, a score of 7 is returned.

[0052] } else { / / If it's a low-priority task score += node.lowUsageResource weight_2 - node.highUsageResource weight_3 / / Same as above } return score / / Returns the score } W_2 and W_3 are used to adjust the weights of resources with the same priority and resources with different priorities. Based on experience, these weights are both set to 1 on the cloud computing platform used in this application.

[0053] Step 403: When the idle node resources in the cluster resources do not meet the needs of the tasks to be processed, the tasks with the same priority in the tasks to be processed are configured in the same node in the cluster resources to reduce the resource fragmentation rate in the cluster resources.

[0054] To reduce the number of resource fragments, the resource preemption subprocess can configure pending tasks to cluster nodes with the same priority but more resource consumption. By increasing the number of idle cluster nodes, the number of resource fragments in the cluster can be reduced.

[0055] Step 404: When configuring corresponding cluster resources for the tasks to be processed according to their priority, the resource scheduling subprocess determines whether the idle node resources in the cluster resources meet the requirements of the high-priority tasks in the tasks to be processed.

[0056] The processing of the resource scheduling subprocess can be represented by the following pseudocode: Func calculateScoreSoftPreempt(node, task){ if task.priority == lowPriority return 0 / / Only calculate the highest priority task, because lower priority tasks cannot be preempted (there is no lower priority task that can be preempted by it). The score for a lower priority task is returned as 0. if task.requiredResource>node.highIdleResource || return 0 / / If idle resources do not meet the task requirements, the score is 0. highIdleResource equals idle (idle) + lowUsageResource (resources used in low priority), because resources occupied by idle and low priority can be used by high priority.

[0057] return node.highUsageResource / / The more resources are occupied by the same priority, the higher the score. Here, because low-priority tasks can be preempted, the amount of low-priority resources is not included in the calculation. Therefore, this reduces resource fragmentation in the cluster, improving resource utilization efficiency, while ensuring timely processing of high-priority tasks.

[0058] Step 405: When the idle node resources in the cluster resources meet the requirements of the high-priority tasks in the pending tasks, the high-priority tasks in the pending tasks are processed through the idle node resources.

[0059] Step 406: When the idle node resources in the cluster resources do not meet the needs of high-priority tasks in the pending tasks, the tasks with the same priority in the high-priority tasks are configured in the same node in the cluster resources to reduce the resource fragmentation rate in the cluster resources.

[0060] In some embodiments of the present invention, when the resource scheduling mode is a forced high-low affinity mode, the cluster resources carrying high-priority task nodes are determined; the cluster resources carrying high-priority task nodes are traversed, and when a high-priority task node carries a low-priority task, the high-priority task among the pending tasks preempts the task node carrying the low-priority task.

[0061] The calculation of the forced high-low affinity mode score can be expressed by Formula 3: Formula 3 The forced high-low affinity mode can also be represented by pseudocode as follows: Func calculateScoreHard(node, task){ if task.requiredResource>node.highIdleResource return 0 / / If available resources are insufficient to meet the task requirements, the score is 0. return node.highUsageResource / node.capacity The amount of resources on node W_1 used by high-priority tasks is used as the node's score. The more resources are occupied by tasks of the same priority, the higher the score.

[0062] } Func getNodeWithHighestScore(task, nodes){ / / Get the node with the highest score max_node = NULL max_score = 0 for node in nodes{ score = calculateScoreHard(node, task) if score>max_score{ max_score = score max_node = node } } return node } Func allocate(task){ node = getNodeWithHighestScore(task,nodes) if node.allocate(task) == failed { / / If the highest-scoring node has a low-priority task, preempt it. node.preempt(task) } node.allocate(task) / / After the preemption is completed, the remaining resources are allocated to the high-priority tasks waiting to be scheduled.

[0063] } W_1 is used to adjust the weights of the high-low affinity algorithm. Based on experience, the weights on the cloud computing platform used in this application are set to 1. Therefore, forcing the high-low affinity mode ensures that the card fragmentation problem of high-priority tasks is completely unaffected by the scheduling of low-priority tasks.

[0064] refer to Figure 5 , Figure 5This is a schematic diagram illustrating the testing process of a resource scheduling method in an embodiment of the present invention. The resource scheduling method provided in this application is tested below. The test environment cluster has two nodes, each with eight resources. Evaluation metrics include: the amount of resources that failed to create due to resource fragmentation: when a high-priority task fails to create due to resource fragmentation, this metric is added to the total resource amount required for that task. This directly reflects how many tasks fail due to fragmentation. The lower this value, the better the algorithm. The average amount of low-priority and idle resources in the cluster when resource fragmentation fails. This is because low-priority and idle resources are resources that high-priority tasks can use. If there are many idle resources when failure occurs, the algorithm is poor. The lower this value, the better the algorithm.

[0065] The test results are shown in Table 1.

[0066] Table 1 In binpack mode, node 1 has two tasks with different priorities, one high and one low, each occupying 4 resources. The high-priority task awaiting scheduling will be scheduled to node 2. If another high-priority task with 8 resources is then assigned, it cannot be scheduled. Although there are 4 idle resources and 4 low-priority resources on the two nodes, they are distributed across the two nodes.

[0067] For the mild high-low affinity mode, we try to avoid placing high and low priority tasks on the same node. Therefore, high and low priority tasks are distributed on nodes 1 and 2 respectively. Four high priority tasks are distributed and scheduled to node 1. If eight more high priority tasks are then distributed, they can be scheduled to node 2 to preempt low priority tasks.

[0068] However, the mild high-low affinity mode cannot completely guarantee that high and low priority items will not be on the same node, and fragmentation issues can still occur. The forced high-low affinity mode, on the other hand, guarantees that high priority items will not be fragmented due to low priority items. When four high priority items are distributed, they will be forcibly placed on node 1, even if node 2 is idle. If eight more high priority items are then distributed, they can be scheduled to node 2.

[0069] In some embodiments of the present invention, the tasks to be processed can be parsed to obtain their priority identifiers; the priorities of the received tasks to be processed can be sorted according to their priority identifiers; and a corresponding task queue can be created according to the priorities of the tasks to be processed. This enables differentiated processing for different customers. Simultaneously, the task queue is traversed to determine the highest priority task; the link quality of each link in the network resources is determined; and the link with the highest link quality in the network resources is configured for the highest priority task to enable transmission of tasks in the task queue through the configured link. This allows for flexible determination of resource scheduling modes that match the cluster resources based on the resource fragmentation rate. By using a forced high-low affinity mode and a mild high-low affinity mode, the resource fragmentation rate in the cluster resources can be reduced, improving resource utilization efficiency, while ensuring timely processing of high-priority tasks, guaranteeing data processing speed for cloud server users, and improving user experience.

[0070] The following uses the cluster resource manager as an example of the WeChat server's resource manager to illustrate the resource scheduling method involved in this invention, wherein, in conjunction with Figure 1 The diagram illustrates the usage environment of the resource scheduling method according to an embodiment of the present invention. Terminals (including terminals 10-1 and 10-2) are equipped with corresponding clients capable of performing different functions. Each client (including terminals 10-1 and 10-2) obtains different information from the corresponding cloud server 200 via network 300 through the WeChat application for browsing. The terminals connect to the cloud server 200 via network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both, using a wireless link for data transmission. The cloud server 200 runs a cluster resource manager compatible with the WeChat application to schedule resources. Each machine carries 8 resources for receiving configuration from the cluster resource manager, and the resource fragmentation rate threshold is 0.8. For the resource scheduling process of the WeChat server's resource manager, refer to... Figure 6 , Figure 6 The following is an optional flowchart illustrating the resource scheduling method provided in an embodiment of the present invention. Figure 6 The steps shown are explained.

[0071] Step 601: Receive tasks to be processed through different service cluster front-ends of the cloud computing server.

[0072] Step 602: The cluster resource manager obtains the tasks to be processed and determines the relationship between the resource fragmentation rate and the resource fragmentation rate threshold in the cluster resources.

[0073] Step 603: Determine whether the resource fragmentation rate in the cluster resources is greater than the resource fragmentation rate threshold. If yes, proceed to step 604; otherwise, proceed to step 605.

[0074] Step 604: Trigger the mild high-low affinity mode.

[0075] Step 605: Trigger the forced high-low affinity mode.

[0076] Step 606: Traverse the queue of tasks to be processed, determine the task with the highest priority, and configure the link with the highest link quality in the network resources.

[0077] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A resource scheduling method, characterized in that, The method includes: Receive the tasks to be processed and submit them to the cluster resource manager; The cluster resource manager is used to determine the resource fragmentation rate in the cluster resources; Based on the resource fragmentation rate in the cluster resources, a resource scheduling mode matching the cluster resources is determined, wherein the resource scheduling mode includes: a forced high-low affinity mode and a mild high-low affinity mode. When the resource scheduling mode is the mild high-low affinity mode, the resource scheduling subprocess of the mild high-low affinity mode is triggered, and the resource scheduling subprocess configures the corresponding cluster resources for the task to be processed according to the priority of the task to be processed; or the resource preemption subprocess of the mild high-low affinity mode is triggered, and the resource preemption subprocess configures the corresponding cluster resources for the task to be processed according to the priority of the task to be processed. When the resource scheduling mode is the forced high-low affinity mode, the node resources carrying high-priority tasks in the cluster resources are traversed. When the node resources are carrying low-priority tasks at the same time, the high-priority task among the pending tasks will preempt the task node carrying the low-priority task.

2. The method of claim 1, wherein, The step of receiving tasks to be processed and submitting the tasks to be processed to the cluster resource manager includes: Tasks to be processed are received from the front end of different service clusters of cloud computing servers; The system access components in the different service clusters are triggered, and the tasks to be processed are submitted to the cluster resource manager that matches the different service clusters through the system access components in the different service clusters.

3. The method of claim 1, wherein, The step of determining a resource scheduling mode that matches the cluster resources based on the resource fragmentation rate in the cluster resources includes: Determine the resource fragmentation rate threshold that matches the cluster resource manager; When the resource fragmentation rate in the cluster resources is greater than the resource fragmentation rate threshold, the resource scheduling mode that matches the cluster resources is determined to be the mild high-low affinity mode. When the resource fragmentation rate in the cluster resources is less than or equal to the resource fragmentation rate threshold, the resource scheduling mode matching the cluster resources is determined to be the forced high-low affinity mode.

4. The method of claim 3, wherein, The method further includes: Trigger a monitoring process that matches the cluster resource manager; The resource fragmentation rate in the cluster resources is monitored in real time through a monitoring process that matches the cluster resource manager. When the resource fragmentation rate in the cluster resources reaches the resource fragmentation rate threshold, the resource scheduling mode that matches the cluster resources is dynamically adjusted.

5. The method according to claim 1, characterized in that, The resource scheduling subprocess that triggers the mild high-low affinity mode configures corresponding cluster resources for the tasks to be processed according to their priorities, including: Determine whether the idle node resources in the cluster resources meet the requirements of the task to be processed; When the idle node resources in the cluster resources meet the task to be processed, the task to be processed is processed through the idle node resources. When the idle node resources in the cluster resources are insufficient to meet the tasks to be processed, tasks with the same priority in the tasks to be processed are configured in the same node in the cluster resources to reduce the resource fragmentation rate in the cluster resources.

6. The method according to claim 1, characterized in that, The resource scheduling subprocess that triggers the mild high-low affinity mode configures corresponding cluster resources for the tasks to be processed according to their priorities, including: Determine whether the idle node resources in the cluster resources meet the high-priority tasks in the pending tasks; When the idle node resources in the cluster resources meet the needs of the high-priority tasks in the pending tasks, the high-priority tasks in the pending tasks are processed through the idle node resources. When the idle node resources in the cluster resources are insufficient to meet the needs of high-priority tasks in the pending tasks, tasks with the same priority among the high-priority tasks are configured to the same node in the cluster resources to reduce the resource fragmentation rate in the cluster resources.

7. The method of claim 1, wherein, The method further includes: The task to be processed is parsed to obtain its priority identifier; The priorities of the received tasks to be processed are sorted according to their priority identifiers. Based on the priority of the tasks to be processed, a corresponding queue of tasks to be processed is created.

8. The method of claim 7, wherein, The method further includes: Based on the priority of the tasks to be processed, the corresponding network resource configuration is adjusted. The step of adjusting the corresponding network resource configuration according to the priority of the tasks to be processed includes: Iterate through the queue of tasks to be processed and determine the task with the highest priority. Determine the link quality of each link in the network resources; Configure the highest priority pending task with the highest quality link in the network resources to enable the transmission of pending tasks in the pending task queue through the configured link.

9. A resource scheduling apparatus, characterized by comprising: The device includes: The information transmission module is used to receive tasks to be processed and submit the tasks to be processed to the cluster resource manager; The information processing module is used to determine the resource fragmentation rate in the cluster resources through the cluster resource manager; The information processing module is further configured to determine a resource scheduling mode that matches the cluster resources based on the resource fragmentation rate in the cluster resources, wherein the resource scheduling mode includes: a forced high-low affinity mode and a mild high-low affinity mode. The information processing module is further configured to, when the resource scheduling mode is the mild high-low affinity mode, trigger the resource scheduling subprocess of the mild high-low affinity mode, and configure corresponding cluster resources for the tasks to be processed according to their priority; or trigger the resource preemption subprocess of the mild high-low affinity mode, and configure corresponding cluster resources for the tasks to be processed according to their priority; when the resource scheduling mode is the forced high-low affinity mode, traverse the node resources carrying high-priority tasks in the cluster resources, and when the node resources also carry low-priority tasks, preempt the task nodes carrying low-priority tasks by the high-priority tasks in the tasks to be processed.

10. The apparatus according to claim 9, characterized in that, The information processing module is also used to determine a resource fragmentation rate threshold that matches the cluster resource manager; The information processing module is further configured to determine, when the resource fragmentation rate in the cluster resources is greater than the resource fragmentation rate threshold, the resource scheduling mode that matches the cluster resources is the mild high-low affinity mode. The information processing module is further configured to determine that the resource scheduling mode matching the cluster resources is the forced high-low affinity mode when the resource fragmentation rate in the cluster resources is less than or equal to the resource fragmentation rate threshold.

11. An electronic device, comprising: The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the resource scheduling method according to any one of claims 1 to 8.

12. A computer-readable storage medium storing executable instructions, wherein the instructions, when executed by a processor, cause the processor to perform operations comprising: When the executable instructions are executed by the processor, they implement the resource scheduling method according to any one of claims 1 to 8.

13. A computer program product comprising computer programs or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the resource scheduling method according to any one of claims 1 to 8.

Images