Task processing method and device

By enabling business nodes in the computing cluster to autonomously acquire and prioritize data tasks, the bottleneck problem of control nodes is solved, task processing efficiency and system resilience are improved, and efficient resource utilization and precise task control are achieved.

CN118897718APending Publication Date: 2024-11-05HUAWEI CLOUD COMPUTING TECHNOLOGIES CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310464537.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-04-26
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

In large-scale computing clusters in cloud scenarios, control nodes become a bottleneck for business operations, leading to reduced task processing efficiency. Furthermore, status reporting and network issues of business nodes can easily cause task backlogs and complicate the processing logic of control nodes.

Method used

The business nodes in the computing cluster independently determine the remaining processing resources, autonomously acquire data processing tasks, determine priorities according to the Service Level Agreement (SLA) level, and dynamically adjust the proportion of processing resources to prioritize high-priority tasks, thereby achieving efficient resource utilization and precise task control.

Benefits of technology

It improves the task processing efficiency of the computing cluster, simplifies the task acquisition control logic, enhances system resilience and data replication efficiency, and ensures the processing efficiency and resource utilization of tasks with different priorities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118897718A_ABST
    Figure CN118897718A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a task processing method which is used for improving the processing efficiency of a data processing task. The method provided by the embodiment of the invention comprises the following steps: a service node determines service state information, wherein the service state information comprises the number of tasks being executed by the service node and the to-be-executed task data volume of the service node; and the service node determines residual processing resources of the service node based on the service state information, wherein the residual processing resources comprise the number of processable tasks and the amount of processable task data. And the service node obtains a data processing task according to the remaining processing resources, wherein the data processing task comprises one or more data processing tasks in a global task queue corresponding to the computing cluster.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and in particular to a task processing method and device. Background Art

[0002] As users migrate their businesses to the cloud, business demands in cloud scenarios are gradually increasing, which also brings challenges to computing clusters in handling large-scale data replication. Data replication tasks in cloud scenarios are often processed by large-scale computing clusters. Currently, large-scale computing clusters usually use centralized time control to allocate and schedule tasks executed on each business node. That is, the computing cluster allocates data replication tasks to multiple business nodes through a control node, thereby achieving a balanced data replication business.

[0003] In the current data task processing process, since the business nodes in the large-scale computing cluster passively receive the data replication tasks assigned by the control node, as the large-scale computing cluster increases, the control node of the large-scale computing cluster will become a business bottleneck, resulting in a decrease in the task processing efficiency of the large-scale computing cluster. At the same time, the working status of each business node needs to be reported to the control node in a timely manner and flow control is performed. When the business node has data replication anomalies or network performance problems, it is easy to cause a backlog of replication tasks and make the processing logic of the control node complicated. Summary of the invention

[0004] The embodiments of the present application provide a task processing method and device for improving the task processing efficiency of any computing cluster.

[0005] The first aspect of the embodiment of the present application provides a task processing method, which is applied to a computing cluster, and the computing cluster includes one or more business nodes. The business node can be a computing device or a cloud server. The method can be executed by the computing cluster, or by a component of the computing cluster, such as a processor, chip or chip system of the computing cluster, or by a logic module or software that can realize all or part of the computing cluster function. The method provided in the first aspect of the embodiment of the present application includes: the business node determines business status information, the business status information is used to indicate the processing resource occupancy of the business node, and the business status information includes the number of tasks being executed by the business node and the amount of task data to be executed by the business node. The business node determines the remaining processing resources of the business node based on the business status information, and the remaining processing resources include the number of tasks that can be processed and the amount of task data that can be processed. The business node obtains a data processing task based on the remaining processing resources, and the data processing task includes one or more data processing tasks in the global task queue corresponding to the computing cluster.

[0006] In the embodiment of the present application, each business node in the computing cluster can independently obtain data processing tasks based on the remaining processing resources of the node itself, without the need for a control node to allocate and manage the data processing tasks of each business node, thereby avoiding the control node becoming a bottleneck for the data processing tasks, thereby improving the processing efficiency of the data processing tasks and simplifying the control logic for the business nodes to obtain data processing tasks.

[0007] In one possible implementation, in the process of a business node acquiring a data processing task based on the remaining processing resources, when the remaining processing resources meet the processing resources required by the data processing task, the business node acquires the data processing task. When the remaining processing resources do not meet the processing resources required by the data processing task, the business node suspends acquiring the data processing task. Specifically, when the number of tasks that can be processed by the business node is greater than or equal to a threshold, and the amount of task data that can be processed by the business node is greater than or equal to the task data amount of the data processing task, the business node acquires the data processing task from the global task queue corresponding to the computing cluster. When the number of tasks that can be processed by the business node is less than a threshold, or the amount of task data that can be processed by the business node is less than the task data amount of the data processing task, the business node suspends acquiring the data processing task from the global task queue corresponding to the computing cluster.

[0008] In the embodiment of the present application, the business node obtains the data processing task according to the remaining processing resources, thereby ensuring the processing efficiency of the business node for the data processing task.

[0009] In a possible implementation, the business node determines the priority of executing the data processing task according to the service level agreement SLA level corresponding to the data processing task, and different SLA levels correspond to different priorities. The service level agreement SLA levels corresponding to the data processing task include gold, silver and bronze levels, and the priority of the task includes VIP, advanced, medium and low levels. Among them, the gold level of the service level agreement SLA level corresponds to VIP and advanced, the silver level of the service level agreement SLA level corresponds to medium priority, and the bronze level of the service level agreement SLA level corresponds to low priority.

[0010] In the embodiment of the present application, the business node can determine the priority of executing the data processing task according to the service level agreement SLA level corresponding to the data processing task, so as to process the data processing tasks of different SLA levels according to different priorities. Therefore, the business node has independent service level level control capability, so that the business node can accurately control the business scenario to achieve the service level level.

[0011] In a possible implementation, tasks of different priorities have different proportions of processing resources of the service node, and the processing resources include bandwidth resources and concurrency. Bandwidth resources determine the amount of task data that the service node can process, and concurrency determines the number of tasks that the service node can process.

[0012] In the embodiment of the present application, tasks of different priorities have different proportions of processing resources of the business node, thereby ensuring the processing efficiency of higher priority data processing tasks and providing differentiated processing efficiency for data processing tasks of different priorities.

[0013] In one possible implementation, the service node adjusts the proportion of processing resources of tasks of different priorities to the service node according to the actual proportion of processing resources used by tasks of different priorities to the service node. When there are no high-priority tasks, the service node increases the proportion of processing resources of tasks of medium priority and the proportion of processing resources of tasks of low priority. For example, when there are no high-priority tasks in the task queue, the service node increases the proportion of processing resources of tasks of medium priority and the proportion of processing resources of tasks of low priority.

[0014] In the embodiment of the present application, the business node can dynamically adjust the proportion of data processing tasks of different priorities to processing resources, thereby improving the utilization rate of the processing resources of the business node.

[0015] In one possible implementation, when a business node adjusts the proportion of processing resources used by tasks of different priorities to a business node based on the proportion of processing resources actually used by tasks of different priorities to the business node, when the proportion of processing resources actually used by tasks of high priority to the business node is less than a first threshold, the proportion of processing resources used by tasks of low priority to the business node is increased.

[0016] In the embodiment of the present application, when the actual usage ratio of the processing resources of the business node by the higher priority tasks is relatively small, the business node can increase the usage ratio of the processing resources of the business node by the lower priority tasks, thereby improving the processing efficiency of the lower priority data processing tasks.

[0017] In one possible implementation, when a business node adjusts the proportion of processing resources used by tasks of different priorities to a business node based on the proportion of processing resources actually used by tasks of different priorities to the business node, when the proportion of processing resources actually used by a high-priority task to the business node is greater than a second threshold, the proportion of processing resources used by the high-priority task to the business node is increased until the proportion of processing resources used by the high-priority task to the business node reaches an upper limit.

[0018] In the embodiment of the present application, when the actual usage ratio of the processing resources of the business node by the tasks with higher priority increases, the business node can reduce the usage ratio of the processing resources of the business node by the tasks with lower priority, thereby giving priority to ensuring the processing efficiency of the data processing tasks with higher priority.

[0019] In a possible implementation manner, the data processing service includes a data replication task, and the service node sends a data read / write request or a data write request to the service object according to the data replication task.

[0020] In the embodiment of the present application, the business node can obtain the data replication task, and send a data read or write request or a data write request to the service object according to the data replication task, thereby improving the data replication efficiency of the computing cluster.

[0021] In a possible implementation, the service node includes a first service node and a second service node. When a transmission channel of the first service node fails, the first service node obtains a data processing task through the second service node and processes the data processing task.

[0022] When a service node of a computing cluster in an embodiment of the present application fails, the service node can obtain data processing tasks based on other service nodes in the computing cluster.

[0023] In a possible implementation, multiple service nodes of a computing cluster simultaneously obtain data replication tasks from a global task queue. When the remaining processing resources of all service nodes are lower than a threshold, the computing cluster requests to add a new service node to obtain data replication tasks.

[0024] In the embodiment of the present application, the computing cluster can increase the business nodes to obtain data processing tasks, thereby realizing the linear expansion of the data processing capacity of the computing cluster. Since the business nodes can independently obtain data processing tasks, the expansion of the computing cluster does not affect the work of other business nodes, thereby improving the system resilience.

[0025] The second aspect of the embodiment of the present application provides a task processing device, including an acquisition unit and a processing unit. The processing unit is used to determine business status information, and the business status information includes the number of tasks being executed by the business node and the amount of task data to be executed by the business node. The processing unit is also used to determine the remaining processing resources of the business node based on the business status information, and the remaining processing resources include the number of processable tasks and the amount of processable task data. The acquisition unit is used to acquire data processing tasks based on the remaining processing resources, and the data processing tasks include one or more data processing tasks in the global task queue corresponding to the computing cluster.

[0026] In a possible implementation, the acquisition unit is specifically configured to acquire the data processing task when the remaining processing resources satisfy the processing resources required by the data processing task, and to suspend acquisition of the data processing task when the remaining processing resources do not satisfy the processing resources required by the data processing task.

[0027] In a possible implementation, the processing unit is further configured to determine a priority for executing the data processing task according to a service level agreement SLA level corresponding to the data processing task, and different SLA levels correspond to different priorities.

[0028] In a possible implementation, tasks of different priorities occupy different proportions of processing resources of a service node, and the processing resources include bandwidth resources and concurrency.

[0029] In a possible implementation manner, the processing unit is further configured to adjust the proportion of processing resources of the service node by tasks of different priorities according to the proportion of processing resources actually used by the tasks of different priorities.

[0030] In a possible implementation, the processing unit is specifically configured to increase the proportion of processing resources used by low-priority tasks on the service node when the actual proportion of processing resources used by high-priority tasks on the service node is less than a first threshold.

[0031] In a possible implementation, the processing unit is further configured to increase the proportion of processing resources used by high-priority tasks on the service node when the proportion of actual processing resources used by high-priority tasks on the service node is greater than a second threshold.

[0032] In a possible implementation, the data processing service includes a data replication task, and the device further includes a transceiver unit, which is used to send a data read / write request or a data write request to the service object according to the data replication task.

[0033] In a possible implementation, the service node includes a first service node and a second service node, and the acquisition unit is further configured to acquire the data processing task through the second service node when a transmission channel of the first service node fails.

[0034] A third aspect of an embodiment of the present application provides a computing device, including a processor, the processor is coupled to a memory, the processor is used to store instructions, when the instructions are executed by the processor, the computing device executes the method described in the first aspect or any possible implementation of the first aspect.

[0035] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having instructions stored thereon. When the instructions are executed, the computer executes the method described in the first aspect or any possible implementation of the first aspect.

[0036] A fifth aspect of an embodiment of the present application provides a computer program product, which includes instructions, and is characterized in that when the instructions are executed, the computer implements the method described in the first aspect or any possible implementation method of the first aspect.

[0037] It can be understood that the beneficial effects that can be achieved by any of the task processing devices, computing equipment, computer-readable media or computer program products provided above can refer to the beneficial effects in the corresponding computing system and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of a system architecture of a computing cluster provided in an embodiment of the present application;

[0039] Figure 2 A flowchart of a task processing method provided in an embodiment of the present application;

[0040] Figure 3 A flowchart of another task processing method provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of a data processing task for obtaining data provided in an embodiment of the present application;

[0042] Figure 5 A flowchart of another task acquisition processing method provided in an embodiment of the present application;

[0043] Figure 6 A schematic diagram of the structure of a task processing device provided in an embodiment of the present application;

[0044] Figure 7 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0045] Figure 8 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0046] Fig. 9 A schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The embodiments of the present application provide a task processing method and device for improving the task processing efficiency of a computing cluster.

[0048] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0049] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0050] The task processing method and device provided in the embodiments of the present application are described below in conjunction with the accompanying drawings.

[0051] See also Figure 1 , Figure 1 A schematic diagram of a system architecture diagram used in a task processing method provided in an embodiment of the present application. Figure 1 In the system architecture shown, the computing cluster 100 includes one or more service nodes 101, and the service node 101 includes a task acquisition module 1011, a data replication module 1012, a data docking module 1013, a quality of service QoS control module 1014 and a data transmission module 1015. The specific functions of each module in the computing cluster 100 are described below.

[0052] The computing cluster 100 includes one or more business nodes 100, wherein the business node 101 is used to obtain data processing tasks according to the business node's own state information and execute data processing tasks, wherein the data processing tasks include data replication tasks. The computing cluster 100 in the embodiment of the present application can be a physical cluster composed of servers, or a virtual cluster composed of virtual machines or containers, without specific limitation.

[0053] The service node 101 includes a task acquisition module 1011, a data replication module 1012, a data docking module 1013, a quality of service QoS control module 1014 and a data transmission module 1015. The task acquisition module 1011 is used to obtain data processing tasks from the global task queue. The task acquisition module 1011 is also used to perform task-level QoS control on the service node 101, including controlling the concurrency of the data processing tasks of the service node 101 and the data volume of the data processing tasks.

[0054] The data replication module 1012 is used to split the data processing tasks and execute the data processing tasks. The data replication module 1012 generates data streams in the process of executing the data processing tasks, and the data streams include read data streams and write data streams. The data docking module 1013 is used to provide a data read and write interface. The business node 101 reads or writes data from the storage node corresponding to the business node through the data read and write interface. The storage node is, for example, an object bucket and file storage system based on a simple storage service (S3).

[0055] The QoS control module 1014 is used to perform data-level QoS control on the service node 101. Specifically, the QoS control module 1014 controls the data flow and data concurrency of data transmission during the process of the service node 101 executing the data processing task.

[0056] The data transmission module 1015 is used for the business node 101 to perform data transmission across availability zones (AZ) or regions. For example, during the process of data transmission across availability zones or regions performed by the business node 101, the data transmission module 1015 sends data generated by executing data processing tasks to business nodes in other computing clusters across availability zones.

[0057] It should be noted that the task processing method provided in the embodiments of the present application and Figure 1 The system architecture shown can be applied to various application scenarios such as data backup, data disaster recovery, data migration, data cloning and data distribution, without specific limitation.

[0058] See also Figure 2 , Figure 2 A flowchart of a task processing method is provided for the implementation of this application. Figure 2 In the example shown, the task processing method provided in the embodiment of the present application includes the following steps:

[0059] 201. The service node determines service status information.

[0060] The service node obtains service status information, which is used to indicate the processing resource usage of the service node. The service status information includes the number of tasks being executed by the service node and the amount of task data of the tasks to be executed by the service node. The service node can calculate the number of tasks that can be processed by the service node based on the number of tasks being executed. The service node can also calculate the number of tasks that can be executed by the service node based on the amount of task data of the tasks to be executed.

[0061] In a possible implementation, when a service node executes a data replication task, the service information acquired by the service node includes the number of data replication tasks being executed by the service node and the amount of data of the replication tasks to be executed. Each service node in the computing cluster can acquire its own service status information.

[0062] See also Figure 3 , Figure 3 A flow chart of a service node performing a replication task provided in an embodiment of the present application. Figure 3 In the example shown, the business nodes in computing cluster 1 are used to perform data replication tasks. The business nodes in computing cluster 1 need to replicate data from source nodes to destination nodes. The destination nodes are connected to the business nodes in computing cluster 2. Computing cluster 2 and computing cluster 1 belong to different regions.

[0063] exist Figure 3 In the example shown, before the business node in computing cluster 1 obtains the data replication task, the business node obtains the business status information of the business node itself, and the business status information includes the number of data replication tasks being processed and the data volume of the data replication tasks to be processed. For example, the business status information obtained by the business node includes that the number of replication tasks being processed by the business node is 3, and the total data volume of the data replication tasks to be processed is 3M.

[0064] 202. The service node determines the remaining processing resources of the service node based on the service status information.

[0065] The service node determines the remaining processing resources of the service node based on the service status information, and the remaining processing resources include the number of tasks that can be processed by the service node and the amount of task data that can be processed by the service node. Specifically, the service node determines the number of tasks that can be processed based on the number of tasks that can be executed concurrently by the service node and the number of tasks being executed. The service node can also determine the amount of task data that can be processed by the service node based on the total amount of task data that can be executed by the service node and the total amount of task data to be processed.

[0066] Please continue reading Figure 3 ,exist Figure 3In the example shown, after obtaining the business status information of the business node itself, the business node in computing cluster 1 determines the remaining processing resources of the business node based on the business status information, and the remaining processing resources include the number of tasks that can be processed by the business node and the amount of task data that can be processed by the business node. For example, the remaining processing resources calculated by the business node include the number of tasks that can be processed by the business node is 1, and the amount of task data that can be processed by the business node is 2M.

[0067] exist Figure 3 In the example shown, in the process of determining the remaining processing resources of the service node based on the service status information, the service node determines the number of tasks that can be processed based on the number of tasks that the service node can execute concurrently and the number of tasks being executed. For example, if the number of tasks that the service node can execute concurrently is 4 and the number of tasks that the service node is executing is 3, the service node can calculate that the number of tasks that the service node can process is 1.

[0068] exist Figure 3 In the example shown, the service node can also determine the amount of task data that the service node can process based on the total amount of task data that the service node can execute and the total amount of task data to be processed. For example, if the total amount of data that the service node can process is 5M and the total amount of task data to be processed is 3M, the service node can calculate that the amount of task data that the service node can process is 2M.

[0069] 203. The business node obtains data processing tasks according to the remaining processing resources.

[0070] The business node obtains data processing tasks based on the remaining processing resources. When the remaining processing resources of the business node meet the processing resources required by the data processing task, the business node obtains the data processing task from the global task queue corresponding to the computing cluster. When the remaining processing resources do not meet the processing resources required by the data processing task, the business node suspends obtaining the data processing task.

[0071] Specifically, when the number of processable tasks of the service node is greater than or equal to the threshold, and the amount of processable task data of the service node is greater than or equal to the task data amount of the data processing task, the service node obtains the data processing task from the global task queue corresponding to the computing cluster. When the number of processable tasks of the service node is less than the threshold, or the amount of processable task data of the service node is less than the task data amount of the data processing task, the service node suspends obtaining the data processing task from the global task queue corresponding to the computing cluster.

[0072] It is understandable that the processable task data volume and the processable task quantity of the service node in the embodiment of the present application adjust the data processing tasks with inconsistent data volumes. For example, the service node reserves a portion of the processable task data volume for processing tasks with small data volumes, thereby avoiding tasks with large data volumes blocking tasks with small data volumes. The service node reserves a portion of the processable task quantity for processing tasks with large data volumes, thereby increasing the concurrency of tasks with small data volumes without blocking tasks with large data volumes.

[0073] In a possible implementation, when a service node executes a data replication task, the service node obtains the data processing task according to the remaining processing resources. Specifically, the service node obtains the data replication task from the global task queue corresponding to the computing cluster according to the remaining processing resources. When the remaining processing resources of the service node meet the processing resources required for the data processing task, the service node obtains the data replication task from the global task queue corresponding to the computing cluster. When the remaining processing resources do not meet the processing resources required for the data processing task, the service node suspends obtaining the data replication task.

[0074] It should be noted that each service node in the computing cluster can independently obtain data processing tasks from the global task queue. After the service node completes the data processing task and releases the remaining processing resources, the service node obtains data processing tasks from the global task queue again according to the remaining processing resources.

[0075] In the embodiment of the present application, the business node can independently obtain the data replication task according to the remaining processing resources, thereby ensuring the processing efficiency of the business node for the data replication task.

[0076] In a possible implementation, after the service node obtains the data processing task, it splits the data processing task, adds the split data processing task to the task queue of the service node, and executes the data processing task according to the task order in the task queue. For example, after the service node obtains the data replication task from the global task queue, it splits the data replication task into data replication subtasks executed by the service node, adds the data replication subtask to the task queue of the service node, and the service node executes the data replication subtask.

[0077] Please continue reading Figure 3 ,exist Figure 3In the example shown, the business node obtains data processing tasks based on the remaining processing resources. For example, the business node calculates that the number of processable tasks is 1, and the processable task data volume of the business node is 2M, the data volume of the data processing task is 1M, the processable task data volume of the business node is greater than the data processing task, and the business node obtains the data processing task from the global task queue. For another example, the business node calculates that the number of processable tasks is 0, or the processable task data volume of the business node is 2M, the data volume of the data processing task is 3M, the processable task data volume of the business node is less than the data volume of the data processing task, and the business node suspends obtaining data processing tasks from the global task queue.

[0078] See also Figure 4 , Figure 4 A flow chart of a service node acquiring and executing a data processing task provided in an embodiment of the present application. Figure 4 In the example shown, each service node in the computing cluster independently obtains data replication tasks from the global task queue based on the remaining resources of the node. After obtaining the data replication tasks, each service node in the computing cluster splits the data replication tasks to generate data replication subtasks, and the service nodes execute the data replication subtasks. For example, when a service node in the computing cluster executes an independent data replication subtask, the service node writes data to a storage node.

[0079] exist Figure 4 In the example shown, each business node in the computing cluster independently manages processing resources and independently decides whether to obtain data replication tasks based on remaining processing, without reporting business status information and remaining resource information to other nodes.

[0080] In one possible implementation, when a business node executes a data replication task, the business node sends a data read or write request or a data write request to a service object according to the data replication task. The service object includes a source storage node and a destination storage node, such as an object bucket and a file storage system of a simple storage service S3.

[0081] Please continue reading Figure 3 ,exist Figure 3 In the example shown, during the process of the business nodes in computing cluster 1 executing the data replication task, the business nodes in computing cluster 1 send a data read request to the source node based on the data replication task, the business nodes in computing cluster 1 read the replicated data from the source node according to the data read request, and send the replicated data to the business nodes in computing cluster 2. After receiving the replicated data, the business nodes in computing cluster 2 send a data write request to the destination node to write the replicated data to the destination node.

[0082] In a possible implementation, the computing cluster includes a first service node and a second service node. When a transmission channel of the first service node fails, the first service node obtains a data processing task through the second service node and processes the data processing task.

[0083] Please continue reading Figure 4 ,exist Figure 4 In the example shown, after the data transmission channel of business node 1 in the computing cluster fails, business node 1 obtains the data replication task from the global task queue through business node 2. Specifically, business node 1 sends the remaining resource information to business node 2, and business node 2 obtains the data processing task based on the remaining resource information of business node 1, and sends the data processing task to business node 1.

[0084] In a possible implementation, after the service node obtains the data processing task, it determines the priority of executing the data processing task according to the service level agreement SLA level corresponding to the data processing task, and different SLA levels correspond to different priorities. The service level agreement SLA level corresponding to the data processing task includes gold level, silver level and bronze level, and the priority of the task queue includes VIP level, high level, normal level and low level.

[0085] In one possible implementation, the gold level of the service level agreement SLA level corresponds to VIP task queues and senior task queues, the silver level of the service level agreement SLA level corresponds to medium priority task queues, and the bronze level of the service level agreement SLA level corresponds to low priority task queues.

[0086] In the embodiment of the present application, tasks of different priorities have different proportions of processing resources for the business node. The processing resources include bandwidth resources and concurrency. The concurrency is the number of data processing tasks concurrently executed by the business node, and the bandwidth resources include the amount of data processed by the business node per unit time.

[0087] See also Figure 5 , Figure 5 A schematic diagram of QoS control of a service node of a computing cluster provided in an embodiment of the present application. Figure 5 In the example shown, there are multiple different tasks in the global task queue, and different tasks have different priorities. For example, the priorities of tasks include VIP priority, high priority, normal priority, and low priority.

[0088] exist Figure 5In the example shown, tasks of different priorities account for different proportions of processing resources of business nodes. For example, VIP priority tasks account for 100% of processing resources of business nodes, high priority tasks account for 50% of processing resources of business nodes, normal priority tasks account for 30% of processing resources of business nodes, and low priority tasks account for 20% of processing resources of business nodes.

[0089] In one possible implementation, the service node can dynamically adjust the proportion of tasks of different priorities to the processing resources of the service node. Specifically, the service node adjusts the proportion of tasks of different priorities to the processing resources of the service node according to the actual proportion of processing resources used by tasks of different priorities to the processing resources of the service node. For example, when there are no high-priority tasks in the task queue, the service node increases the proportion of medium-priority tasks to the processing resources and the proportion of low-priority tasks to the processing resources.

[0090] Please continue reading Figure 5 ,exist Figure 5 In the example shown, the three business nodes in the computing cluster obtain data replication tasks from the global task queue. The high-priority tasks of each business node account for 50% of the concurrent tasks of the business node, the normal-priority tasks account for 30% of the concurrent tasks of the business node, and the low-priority tasks account for 20% of the concurrent tasks of the business node. Assuming that each business node can obtain 10 concurrent tasks, the high-priority tasks can account for 5, the normal-priority tasks can account for 3, and the low-priority tasks can account for 2.

[0091] exist Figure 5 In the example shown, business nodes 1, 2, and 3 in the computing cluster obtain data replication tasks from the global task queue in turn. Business node 1 obtains 5 high priority tasks A1, B1, C1, D1, and E1, 3 normal priority tasks A2, B2, and C2, and 2 low priority tasks A3 and B3. Business node 2 obtains 5 high priority tasks F1, G1, H1, I1, and J1, 3 normal priority tasks D2, E2, and F2, and 2 low priority tasks D3 and E3. When business node 3 obtains data replication tasks from the global task queue, there are only two high priority tasks K1 and L1. Business node 3 increases the proportion of normal priority tasks in processing resources. Business node 3 obtains 2 high priority tasks K1 and L1, 6 normal priority tasks G2, H2, I2, J2, K2, L2, and 2 low priority tasks E3 and F3.

[0092] In one possible implementation, when the actual usage ratio of the processing resources of the service node by the high-priority task is less than the first threshold, the service node reduces the usage ratio of the processing resources of the service node by the high-priority task and increases the usage ratio of the processing resources of the service node by the low-priority task. When the actual usage ratio of the processing resources of the service node by the high-priority task is greater than the second threshold, the service node increases the usage ratio of the processing resources of the service node by the high-priority task and reduces the usage ratio of the processing resources of the service node by the low-priority task.

[0093] Please continue reading Figure 5 ,exist Figure 5 In the example shown, the business node adjusts the proportion of processing resources of tasks of different priorities to the business node according to the actual proportion of processing resources used by tasks of different priorities to the business node. For example, within a period of time, when the actual proportion of processing resources used by tasks of high priority to the business node is less than 40%, the business node adjusts the proportion of processing resources used by tasks of high priority to the business node from 50% to 40%, and increases the proportion of processing resources used by tasks of normal priority to the business node from 30% to 40%.

[0094] For another example, over a period of time, when the actual usage ratio of high-priority tasks to the processing resources of the business node is greater than or equal to 40%, the business node adjusts the usage ratio of high-priority tasks to the processing resources of the business node from 40% to 50%, and the business node adjusts the usage ratio of normal-priority tasks to the processing resources of the business node from 40% to 30%.

[0095] In one possible implementation, multiple business nodes of the computing cluster simultaneously obtain data replication tasks from the global task queue. When the remaining processing resources of all business nodes are lower than a threshold, the computing cluster requests to add new business nodes to obtain data replication tasks, thereby achieving linear expansion of the computing cluster.

[0096] It can be seen from the above embodiments that each business node in the computing cluster in the embodiments of the present application can independently obtain data processing tasks based on the remaining processing resources of the node itself, which simplifies the control logic of the business node obtaining data processing tasks, thereby improving the processing efficiency of data processing tasks.

[0097] The above introduces the task processing method provided by the embodiment of the present application. The following introduces the task processing device provided by the embodiment of the present application.

[0098] See also Figure 6 , Figure 6 A schematic diagram of the structure of a task processing device provided in an embodiment of the present application. Figure 6In the example shown, the task processing device is used to implement the various steps executed by the computing cluster in the above embodiments. The task processing device 600 includes an acquisition unit 601 , a processing unit 602 , and a transceiver unit 603 .

[0099] The processing unit 602 is used to determine the business status information, which includes the number of tasks being executed by the business node and the amount of task data to be executed by the business node. The processing unit 602 is also used to determine the remaining processing resources of the business node based on the business status information, and the remaining processing resources include the number of processable tasks and the amount of processable task data. The acquisition unit 601 is used to acquire data processing tasks according to the remaining processing resources, and the data processing tasks include one or more data processing tasks in the global task queue corresponding to the computing cluster.

[0100] In a possible implementation, the acquisition unit 601 is specifically configured to acquire the data processing task when the remaining processing resources meet the processing resources required by the data processing task, and to suspend acquisition of the data processing task when the remaining processing resources do not meet the processing resources required by the data processing task.

[0101] In a possible implementation, the processing unit 602 is further configured to determine the priority of executing the data processing task according to a service level agreement SLA level corresponding to the data processing task, and different SLA levels correspond to different priorities.

[0102] In a possible implementation, tasks of different priorities occupy different proportions of processing resources of a service node, and the processing resources include bandwidth resources and concurrency.

[0103] In a possible implementation, the processing unit 602 is further configured to adjust the proportion of processing resources of the service node by tasks of different priorities according to the proportion of processing resources actually used by the tasks of different priorities.

[0104] In a possible implementation, the processing unit 602 is specifically configured to increase the proportion of processing resources used by low-priority tasks on the service node when the actual proportion of processing resources used by high-priority tasks on the service node is less than a first threshold.

[0105] In a possible implementation, the processing unit 602 is further configured to increase the proportion of processing resources used by the high-priority tasks on the service node when the proportion of actual use of processing resources of the service node by the high-priority tasks is greater than a second threshold.

[0106] In a possible implementation, the data processing service includes a data replication task, and the device further includes a transceiver unit 603, and the transceiver unit 603 is used to send a data read / write request or a data write request to the service object according to the data replication task.

[0107] In a possible implementation, the service node includes a first service node and a second service node, and the acquisition unit 601 is further configured to acquire a data processing task through the second service node when a transmission channel of the first service node fails.

[0108] It should be understood that the division of the units in the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And the units in the device can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some units can also be implemented in the form of software calling through processing elements, and some units can be implemented in the form of hardware. For example, each unit can be a separately established processing element, or it can be integrated in a certain chip of the device. In addition, it can also be stored in the memory in the form of a program, and called and executed by a certain processing element of the device. The function of the unit. In addition, all or part of these units can be integrated together, or they can be implemented independently. The processing element described here can also be a processor, which can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each unit above can be implemented by an integrated logic circuit of hardware in the processor element or in the form of software calling through a processing element.

[0109] It is worth noting that, for the above method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required for the present invention.

[0110] Other reasonable step combinations that can be thought of by those skilled in the art based on the above description also fall within the scope of protection of the present invention. Secondly, those skilled in the art should also be familiar with the fact that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present invention.

[0111] See also Figure 7 , Figure 7 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. Figure 7 As shown, the computing device 700 includes: a processor 701, a memory 702, a communication interface 703 and a bus 704. The processor 701, the memory 702 and the communication interface 703 are coupled via a bus (not marked in the figure). The memory 702 stores instructions. When the execution instructions in the memory 702 are executed, the computing device 700 executes the method executed by the cloud server in the above method embodiment.

[0112] The computing device 700 may be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA), or a combination of at least two of these integrated circuit forms. For another example, when the unit in the device can be implemented in the form of a processing element scheduler, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call a program. For another example, these units can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0113] The processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

[0114] The memory 702 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0115] The memory 702 stores executable program codes, and the processor 701 executes the executable program codes to respectively implement the functions of the aforementioned acquisition unit, processing unit, and transceiver unit, thereby implementing the aforementioned task processing method. That is, the memory 702 stores instructions for executing the aforementioned task processing method.

[0116] The communication interface 703 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 700 and other devices or a communication network.

[0117] In addition to the data bus, the bus 704 may also include a power bus, a control bus, a status signal bus, etc. The bus may be a peripheral component interconnect express (PCIe) bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The bus may be divided into an address bus, a data bus, a control bus, etc.

[0118] See also Figure 8 , Figure 8 A schematic diagram of a computing device cluster provided in an embodiment of the present application. Figure 8 As shown, the computing device cluster 800 includes at least one computing device 700 .

[0119] like Figure 8 As shown, the computing device cluster 800 includes at least one computing device 700. The memory 702 in one or more computing devices 700 in the computing device cluster 800 may store the same instructions for executing the above-mentioned task processing method.

[0120] In some possible implementations, the memory 702 of one or more computing devices 700 in the computing device cluster 800 may also store partial instructions for executing the above-mentioned task processing method. In other words, the combination of one or more computing devices 700 may jointly execute instructions for executing the above-mentioned task processing method.

[0121] It should be noted that the memory 702 in different computing devices 700 in the computing device cluster 800 can store different instructions, which are respectively used to execute part of the functions of the above-mentioned task processing device. That is, the instructions stored in the memory 702 in different computing devices 700 can implement the functions of one or more modules.

[0122] In some possible implementations, one or more computing devices 700 in the computing device cluster 800 may be connected via a network, which may be a wide area network or a local area network.

[0123] See also Fig. 9 , Fig. 9 A schematic diagram of computer devices in a computer cluster connected via a network provided in an embodiment of the present application. Fig. 9 As shown, two computing devices 700A and 700B are connected via a network. Specifically, they are connected to the network via a communication interface in each computing device.

[0124] In a possible implementation, the memory in the computing device 700A stores instructions for executing the functions of the acquisition unit and the processing unit. Meanwhile, the memory in the computing device 700B stores instructions for executing the functions of the transceiver unit.

[0125] It should be understood that Fig. 9 The functions of the computing device 700A shown in FIG. 7 may also be completed by multiple computing devices. Similarly, the functions of the computing device 700B may also be completed by multiple computing devices.

[0126] In another embodiment of the present application, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When the processor of the device executes the computer-executable instructions, the device executes the method executed by the cloud server in the above method embodiment.

[0127] In another embodiment of the present application, a computer program product is further provided, the computer program product comprising computer executable instructions stored in a computer readable storage medium. When the processor of the device executes the computer executable instructions, the device executes the method executed by the cloud server in the above method embodiment.

[0128] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0129] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0132] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A task processing method, characterized in that: Applied to a computing cluster, the computing cluster includes one or more service nodes, including: The service node determines service status information, where the service status information includes the number of tasks being executed by the service node and the amount of task data to be executed by the service node; The service node determines the remaining processing resources of the service node based on the service status information, where the remaining processing resources include the number of processable tasks and the amount of processable task data; The service node obtains a data processing task according to the remaining processing resources, where the data processing task includes one or more data processing tasks in a global task queue corresponding to the computing cluster.

2. The method according to claim 1, characterized in that The service node acquiring the data processing task according to the remaining processing resources includes: When the remaining processing resources meet the processing resources required by the data processing task, the service node acquires the data processing task; When the remaining processing resources do not meet the processing resources required by the data processing task, the service node suspends obtaining the data processing task.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The business node determines the priority of executing the data processing task according to the service level agreement SLA level corresponding to the data processing task, and different SLA levels correspond to different priorities.

4. The method according to claim 3, characterized in that The tasks of different priorities have different proportions of processing resources of the service node, and the processing resources include bandwidth resources and concurrency.

5. The method according to claim 4, characterized in that The method further comprises: The service node adjusts the proportion of processing resources of the service node by tasks of different priorities according to the actual proportion of processing resources of the service node used by the tasks of different priorities.

6. The method according to claim 5, characterized in that The adjusting the proportion of processing resources of tasks with different priorities to the service node according to the actual proportion of processing resources used by tasks with different priorities to the service node comprises: When the actual usage ratio of the processing resources of the service node by the high-priority tasks is less than the first threshold, the usage ratio of the processing resources of the service node by the low-priority tasks is increased.

7. The method according to claim 5, characterized in that The adjusting the proportion of processing resources of tasks with different priorities to the service node according to the actual proportion of processing resources used by tasks with different priorities to the service node comprises: When the actual usage ratio of the processing resources of the service node by the high-priority tasks is greater than the second threshold, the usage ratio of the processing resources of the service node by the high-priority tasks is increased.

8. The method according to any one of claims 1 to 7, characterized in that The data processing service includes a data replication task, and the method further includes: The service node sends a data read / write request or a data write request to the service object according to the data replication task.

9. The method according to claim 8, characterized in that The service node includes a first service node and a second service node, and the method further includes: When a transmission channel of the first service node fails, the first service node obtains a data processing task through the second service node.

10. A task processing device, characterized in that: include: A processing unit, configured to determine service status information, wherein the service status information includes the number of tasks being executed by the service node and the amount of task data to be executed by the service node; The processing unit is further used to determine the remaining processing resources of the service node based on the service status information, wherein the remaining processing resources include the number of processable tasks and the amount of processable task data; An acquisition unit is used to acquire a data processing task according to the remaining processing resources, wherein the data processing task includes one or more data processing tasks in a global task queue corresponding to the computing cluster.

11. The device according to claim 10, characterized in that The acquisition unit is specifically used for: When the remaining processing resources meet the processing resources required by the data processing task, obtaining the data processing task; When the remaining processing resources do not meet the processing resources required by the data processing task, the acquisition of the data processing task is suspended.

12. The device according to claim 10 or 11, characterized in that The processing unit is also used for: The priority of executing the data processing task is determined according to the service level agreement SLA level corresponding to the data processing task, and different SLA levels correspond to different priorities.

13. The device according to claim 12, characterized in that The tasks of different priorities have different proportions of processing resources of the service node, and the processing resources include bandwidth resources and concurrency.

14. The device according to claim 13, characterized in that The processing unit is also used for: The proportion of processing resources of the service node used by tasks of different priorities is adjusted according to the actual proportion of processing resources used by tasks of different priorities.

15. The device according to claim 14, characterized in that The processing unit is specifically used for: When the actual usage ratio of the processing resources of the service node by the high-priority tasks is less than the first threshold, the usage ratio of the processing resources of the service node by the low-priority tasks is increased.

16. The device according to claim 15, characterized in that The processing unit is also used for: When the actual usage ratio of the processing resources of the service node by the high-priority tasks is greater than the second threshold, the usage ratio of the processing resources of the service node by the high-priority tasks is increased.

17. The device according to any one of claims 10 to 16, characterized in that The data processing service includes a data replication task, and the device also includes a transceiver unit, and the transceiver unit is used to: Send data read / write request or data write request to the service object according to the data replication task.

18. The device according to claim 17, characterized in that The service node includes a first service node and a second service node, and the acquisition unit is further used for: When a transmission channel of the first service node fails, a data processing task is obtained through the second service node.

19. A computing device, characterized in that The cloud server comprises a processor coupled to a memory, wherein the processor is used to store instructions. When the instructions are executed by the processor, the cloud server executes the method according to any one of claims 1 to 9.

20. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed, the computer is caused to perform the method according to any one of claims 1 to 9.

21. A computer program product, comprising instructions, characterized in that: When the instructions are executed, the computer implements the method according to any one of claims 1 to 9.