Task processing method and device and storage medium

Abstract

The invention provides a task processing method which can be applied to the technical field of computers. The task processing method comprises the following steps: acquiring a first duration consumed by a target node for executing a plurality of first tasks of a target type in a current time period and a resource type configured on the target node; wherein the resource configured on the target node and used for executing the target type task is determined according to the initial resource allocation strategy; the initial resource allocation strategy indicates initial quantity information of the target node for parallel processing of the target type tasks; adjusting an initial resource allocation strategy based on the first duration and the resource type; and the target node carries out parallel processing on a plurality of second tasks of the target type in a subsequent time period according to the target quantity information indicated by the adjusted resource allocation strategy. The invention further provides task processing equipment and a storage medium.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to the fields of resource management and task scheduling technology, and more specifically to a task processing method, device, and storage medium. Background Art

[0002] With the increasing growth of the data processing volume in distributed systems, especially in the high-concurrency scenarios of business task processing volume, due to non-business tasks, such as data destruction tasks, competing with business tasks for bandwidth, CPU, and memory resources, it is easy to cause delays or interruptions in normal business processing. Summary of the Invention

[0003] In view of the above problems, this application provides a task processing method, device, and storage medium.

[0004] In the first aspect of this application, a task processing method applied to a first processor deployed on any target node in a distributed system is provided, including: obtaining the first duration consumed by the target node for executing multiple first tasks of a target type during the current period and the resource types configured on the target node; wherein, the resources configured on the target node for executing tasks of the target type are determined according to an initial resource allocation policy; the initial resource allocation policy indicates the initial quantity information of the target node for parallel processing of tasks of the target type; adjusting the initial resource allocation policy based on the first duration and resource types; the target node parallelly processes multiple second tasks of the target type during the subsequent period according to the target quantity information indicated by the adjusted resource allocation policy.

[0005] In the second aspect of this application, a task processing method applied to a second processor deployed in a distributed system for scheduling the resource configuration of multiple target nodes is provided, including: obtaining multiple adjustment information of multiple target nodes within a predetermined period; wherein, the multiple adjustment information is obtained based on the task processing method applied to the first processor; adjusting the parallel processing parameters indicated by an adjustment policy associated with the resource type based on the multiple adjustment information, so that each of the multiple target nodes obtains target information according to the adjusted adjustment policy, and the target information indicates the quantity of the target node for parallel processing of tasks of the target type.

[0006] The third aspect of this application provides a task processing device, including: a first processor and a second processor.

[0007] A first processor is configured to receive an initial resource allocation policy from a second processor; obtain a first duration consumed by a target node for executing multiple tasks of a target type during a current period and resource types configured on the target node; wherein the resources configured on the target node for executing tasks of the target type are determined according to the initial resource allocation policy; the initial resource allocation policy indicates the quantity information of tasks of the target type that the target node processes in parallel; adjust the initial resource allocation policy based on the first duration and the resource types; and the target node processes multiple second tasks of the target type in parallel during a subsequent period according to the target quantity information indicated by the adjusted resource allocation policy.

[0008] A second processor is configured to send the initial resource allocation policy to the first processor; obtain adjustment information for adjusting the initial resource allocation policy of multiple target nodes within a predetermined period; and adjust parallel processing parameters indicated by an adjustment policy based on the multiple adjustment information.

[0009] A fourth aspect of the present application further provides a computer-readable storage medium, on which a computer program or instruction is stored, and when the computer program or instruction is executed by a processor, the steps of the above method are implemented.

[0010] A fifth aspect of the present application further provides a computer program product, including a computer program or instruction, and when the computer program or instruction is executed by a processor, the steps of the above method are implemented. Description of the Drawings

[0011] Through the following description of the embodiments of the present application with reference to the drawings, the above content and other objects, features, and advantages of the present application will become clearer. In the drawings:

[0012] Figure 1 An application scenario diagram of a task processing method according to an embodiment of the present application is shown;

[0013] Figure 2 A flowchart of a task processing method applied to a first processor according to an embodiment of the present application is shown;

[0014] Figure 3 A flowchart of statistics on the duration consumed by tasks of a target type according to an embodiment of the present application is shown;

[0015] Figure 4A A schematic diagram of dynamically adjusting a resource allocation policy according to an embodiment of the present application is shown;

[0016] Figure 4B A schematic diagram of dynamically adjusting a resource allocation policy according to another embodiment of the present application is shown;

[0017] Figure 4CShows a schematic diagram of dynamically adjusting a resource allocation strategy according to another embodiment of the present application;

[0018] Figure 5 Shows a task processing method applied to a second processor according to an embodiment of the present application;

[0019] Figure 6 Shows a schematic diagram of adjusting parallel processing parameters indicated by an adjustment strategy associated with a resource type based on multiple adjustment information according to an embodiment of the present application;

[0020] Figure 7 Shows a schematic diagram of adjusting multiple parallel processing parameters indicated by an adjustment strategy based on a policy adjustment frequency and a policy adjustment direction according to another embodiment of the present application.

[0021] Figure 8 Shows a schematic diagram of a task processing device according to an embodiment of the present application;

[0022] Figure 9 Shows a block diagram of an electronic device suitable for implementing a task processing method according to an embodiment of the present application. Detailed implementation manners

[0023] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present application. In the following detailed description, for the sake of explanation, many specific details are set forth in order to provide a comprehensive understanding of the embodiments of the present application. However, it is obvious that one or more embodiments can be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts of the present application.

[0024] The terms used herein are merely for describing specific embodiments and are not intended to limit the present application. The terms "including", "comprising" and the like used herein indicate the presence of the described features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] In the case of using expressions such as "at least one of A, B, and C", generally, it should be interpreted according to the meaning that those skilled in the art usually understand this expression (for example, "a system having at least one of A, B, and C" should include, but not be limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0027] A distributed system is a system composed of a group of computer nodes that communicate through a network and coordinate to complete a common task. For example: a distributed storage system, a distributed virtual machine system in a cloud computing environment, etc.

[0028] Whether it is a distributed storage system or a cloud platform, the number of tasks processed is increasing exponentially. In high-concurrency data scenarios, there are business tasks and non-business tasks, such as: competition for bandwidth, CPU, and memory resources between data destruction tasks, resulting in interruptions or delays in business tasks.

[0029] Taking the data destruction task in a distributed storage system as an example, when facing fluctuations in the processing volume of business tasks, since business tasks and non-business tasks will compete for the bandwidth, CPU, or memory resources of the server. In related examples, static quota methods, token bucket algorithms, feedback control methods, or priority scheduling methods are usually used for resource allocation.

[0030] However, the resource allocation methods provided in related examples for non-business tasks, such as: data destruction tasks, all have certain limitations.

[0031] For example: the fixed quota configured by the static quota method cannot respond to load fluctuations. During high-load periods, a quota of 5 tasks per single node is likely to cause task backlogs, and the average waiting time increases from several minutes to several hours, slowing down the data processing efficiency. During the low period of the number of business tasks, the fixed quota causes computing and storage resources to be idle and cannot be flexibly scheduled to other tasks, resulting in waste of hardware investment.

[0032] For example: in the case of large-scale database updates with the token bucket algorithm, the short-term influx of destruction tasks may far exceed the token generation rate, resulting in the task delay jumping from the millisecond level to the second level, affecting business performance. There are significant performance differences between the RAID groups of HDD and SSD, but the token allocation strategy is not optimized specifically. For example, the SSD RAID group can bear higher concurrency due to its high-speed read and write capabilities, while the HDD RAID group is prone to trigger performance bottlenecks due to insufficient tokens.

[0033] For example, the feedback control method adjusts the task rate by dynamically monitoring the system load (such as CPU utilization). However, the performance of HDD RAID 5 drops significantly when dealing with random read and write operations, while SSD RAID 0 can handle them efficiently. However, the feedback mechanism does not adjust differently according to the media type, resulting in overloading of the HDD group and idle resources in the SSD group. In the cloud storage scenario, when the user requests surge, the system needs several seconds to collect metrics and adjust the rate, during which there may be a sharp drop in performance, affecting the user experience.

[0034] For example, the priority scheduling method needs to configure priorities for different application scenarios, with poor generality. And in some special scenarios, strict deadlines are set for data destruction. Since business tasks are given priority, specific data may not be destroyed on schedule, leading to data security risks.

[0035] In view of this, embodiments of the present application provide a task processing method, including: obtaining the first duration consumed by a target node for executing multiple first tasks of a target type in the current period and the resource type configured on the target node; wherein, the resources configured on the target node for executing tasks of the target type are determined according to an initial resource allocation policy; the initial resource allocation policy indicates the initial quantity information of the target type tasks that the target node can process in parallel; adjusting the initial resource allocation policy based on the first duration and the resource type; and processing multiple second tasks of the target type in parallel in the subsequent period according to the target quantity information indicated by the adjusted resource allocation policy.

[0036] Figure 1 The application scenario diagram of the task processing method according to the embodiments of the present application is shown.

[0037] As Figure 1 shown, the application scenario 100 according to this embodiment may include a first terminal device 1011, a second terminal device 1012, a third terminal device 1013, and a first Web server 1021, a second Web server 1022, and a third Web server 1023 that respectively interact with the first terminal device 1011, the second terminal device 1012, and the third terminal device 1013.

[0038] Users can use the first terminal device 1011, the second terminal device 1012, and the third terminal device 1013 to interact with the first Web server 1021, the second Web server 1022, and the third Web server 1023 respectively through the network to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 1011, the second terminal device 1012, and the third terminal device 1013, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only). The information received and processed by the first Web server 1021, the second Web server 1022, and the third Web server 1023 can all be stored in the distributed storage system 103. Multiple storage servers can be configured in the distributed storage system to store the information of the first Web server 1021, the second Web server 1022, and the third Web server 1023 in a distributed manner or send the information required to be read to the first Web server 1021, the second Web server 1022, and the third Web server 1023.

[0039] The first terminal device 1011, the second terminal device 1012, and the third terminal device 1013 can be various electronic devices with a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop computers, desktop computers, and so on.

[0040] The first Web server 1021, the second Web server 1022, and the third Web server 1023 can be servers that provide various services, such as a background management server that supports the websites browsed by users using the first terminal device 1011, the second terminal device 1012, and the third terminal device 1013 (for example only). The background management server can analyze and process data such as user requests received, and feedback the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0041] It should be noted that the task processing method provided by the embodiments of the present application can generally be executed by the storage servers in the distributed storage system 103.

[0042] It should be understood that Figure 1 the numbers of terminal devices, Web servers, and storage servers in the distributed storage system are merely illustrative. According to the implementation requirements, there can be any number of terminal devices, Web servers, and storage servers in the distributed storage system.

[0043] Based on the Figure 1 scenario described below, the task processing method of the embodiment will be described in detail through Figures 2 to 8 the following.

[0044] Figure 2 The flowchart of a task processing method applied to a first processor according to an embodiment of the present application is shown.

[0045] As Figure 2 shown, the task processing method 200 of this embodiment includes operation S210 to operation S230.

[0046] In operation S210, obtain the first duration consumed by the target node to execute multiple first tasks of the target type in the current period and the resource types configured on the target node.

[0047] In operation S220, adjust the initial resource allocation policy based on the first duration and the resource types.

[0048] In operation S230, the target node processes multiple second tasks of the target type in parallel in the subsequent period according to the target quantity information indicated by the adjusted resource allocation policy.

[0049] In the embodiment of the present application, for a distributed storage system, the target node can be any storage server in the distributed storage system. The resource types configured on the target node can be storage media configured on the storage server, such as: HDD (Hard Disk Drive), SSD (Solid State Drive), NVMe (Non -Volatile Memory express), SCM (Storage - Class Memory), etc.

[0050] In some embodiments, the resources configured on the target node for executing tasks of the target type are determined according to the initial resource allocation policy. The initial resource allocation policy indicates the initial quantity information for the target node to process tasks of the target type in parallel.

[0051] In the embodiment of the present application, the target type tasks can be non - business tasks, such as: data destruction tasks. The initial quantity information can be calculated according to the parallel processing volume of a predetermined maximum data destruction task and the number of nodes.

[0052] For example: the parallel processing volume of the maximum data destruction task is 1000, and the number of target nodes configured on the distributed storage system is 4. It can be determined that the initial quantity information for each target node can be 250.

[0053] In some embodiments, there is resource competition between business tasks and target type tasks during execution. The processing volume of business tasks is affected by the user side and shows irregular fluctuations. Therefore, by obtaining in real time the first duration consumed by a target node in executing multiple first tasks of the target type in the current period, it is possible to finely and dynamically allocate the parallel processing volume for the target type tasks, without affecting the allocation of resources such as bandwidth and computing power for the input and output of business tasks by the target node, further reducing the performance jitter caused by resource preemption in the high-concurrency scenario of business tasks and improving system stability.

[0054] In the embodiments of the present application, the first duration may be the average duration consumed by a target type task for a predetermined number of times, and the first duration is obtained by concurrently processing data destruction tasks according to the initial concurrent processing quantity.

[0055] Figure 3 The flowchart showing the statistics of the duration consumed by the target type tasks according to the embodiments of the present application is shown.

[0056] As Figure 3 shown, the target type tasks to be processed by the target node can be executed in the order of arrival at the target node or in the order of the priorities of the target type tasks. First, at the start of the execution of the target type task, the task start time 301 can be recorded, and at the end of the execution of the target type task, the task end time 302 can be recorded.

[0057] Then, operation S310 is executed to calculate the completion duration. And operation S320 is executed to record the task completion times.

[0058] Next, operation S330 is executed to determine whether the task completion times is equal to α. α can represent a predetermined number of times, for example: it can be 1000 times.

[0059] If so, operation S340 is executed to calculate the average duration. For example: the actual duration of 1000 task completions can be divided by 1000 to obtain the average duration. If not, operation S350 is executed to execute the next task and calculate the completion duration of the next task.

[0060] In some embodiments, multiple RAID (Redundant Arrays of Independent Disks) groups are configured on the target node. The initial quantity information of each RAID group can be further calculated. So that the target node can schedule the target type tasks according to the initial quantity information of each RAID group.

[0061] In some embodiments, the number of RAID groups configured on the target node is also dynamically updated, and the parallel processing volume of each RAID group can be dynamically updated according to the actual number of RAID groups.

[0062] Since the characteristics of different storage media are different. For example, HDD has high throughput but random read / write latency reaching the order of hundreds of milliseconds, which is suitable for sequential access scenarios. Although SSD or NVMe has lower read / write latency, only at the order of 30 milliseconds, there is a write amplification effect, and the actual write volume far exceeds the user's demand. The random read / write latency of SCM can be reduced to the order of nanoseconds, but it is difficult to be deployed on a large scale due to high cost.

[0063] Therefore, when dynamically adjusting the number of concurrent processes of the target type task, it is necessary to clarify the storage media type to further improve resource utilization.

[0064] For example: Two RAID groups can be configured on the target node. One RAID is for SSD and the other is for HDD. Due to the difference in storage media performance, the random read / write latency of SSD is usually 30ms, and the random read / write latency of HDD is usually 100ms.

[0065] Therefore, the average duration consumed for performing the data destruction task on the RAID groups of the same type of storage media can be used to further improve the accuracy of the adjustment result of the dynamic adjustment strategy.

[0066] In some embodiments, the parallel processing volume indicated in the initial resource allocation strategy can be adjusted based on the deviation degree and deviation direction between the first duration and the predetermined latency of the RAID group. The adjustment direction of the parallel processing volume is opposite to the deviation direction, and the adjustment degree can be the same as the deviation degree. The predetermined latency can be determined according to the performance of the RAID group.

[0067] For example: For the RAID group with the storage media being HDD, the parallel processing volume of the data destruction task indicated in the initial resource allocation strategy is 15, and the first duration can be 120ms. The predetermined latency of HDD can be 100ms. It can be determined that the positive deviation degree between the current first duration and the predetermined latency of the RAID group is 12%. Then, correspondingly, the parallel processing volume can be reduced by 12% and adjusted to 13. It can be understood that in the next period, for the RAID group with the storage media being HDD, a maximum of 13 data destruction tasks can be processed in parallel.

[0068] In an embodiment of the present application, based on the actual duration consumed by a target node for real-time processing of target type tasks in the current period, and in combination with the performance characteristics of the resource types configured for the target node, the parallel processing volume for target type tasks in the next period is dynamically adjusted, which at least solves the problems that the target node competes for resources with business type tasks when processing target type tasks, resulting in unstable system performance and business interruption or delay, and achieves the technical effect of optimizing concurrent task allocation and system throughput efficiency based on the performance characteristics of resource types.

[0069] According to an embodiment of the present application, adjusting the initial resource allocation policy based on the first duration and resource type may include the following operations: determining a plurality of adjustment policies associated with the resource type; determining a target adjustment policy associated with the first duration from the plurality of adjustment policies; and adjusting the initial resource allocation policy according to the target adjustment policy.

[0070] In some embodiments, the adjustment policy indicates the association relationship between the execution duration threshold and the parallel processing parameters of the target type tasks. Taking a distributed storage system as an example, adjustment policies corresponding to different storage media may be pre-configured.

[0071] In some embodiments, the execution duration threshold may be a pre-configured threshold related to the resource type. For example, for a certain type of RAID group, the configured execution duration thresholds may include gradient settings such as 100ms, 150ms, 200ms, etc.

[0072] In some embodiments, the parallel processing parameters of the target type tasks may include the parallel processing volume.

[0073] Therefore, the adjustment policy may include the association relationship between the gradient-set execution duration threshold and the parallel processing parameters of the target type tasks. For example: Adjustment policy P1 may be that when the actual duration is greater than 100ms, the corresponding parallel processing volume is 15. Adjustment policy P2 may be that when the actual duration is greater than 150ms, the corresponding parallel processing volume is 10.

[0074] It can be understood that the greater the actual execution duration of the target type tasks, the less resources the current node can allocate for the target type tasks. Therefore, a smaller parallel processing volume needs to be allocated to reduce the impact of the execution of the target type tasks on the execution of the business type tasks.

[0075] For example: When the first duration is 120ms, it can be determined that the target adjustment policy is adjustment policy P1. Therefore, the parallel processing volume of the target type indicated by the initial resource allocation policy can be adjusted to 15.

[0076] To verify the technical effects of the embodiments of the present application, in accordance with the task processing method provided by the embodiments of the present application, tests were conducted on the data destruction tasks in the distributed storage system. The test results showed that the data destruction rate of the SSD - type RAID group increased by 40 - 50%, the read - write stability of the HDD - type RAID group increased by 35%, and the throughput of the distributed storage system increased by 22 - 23%.

[0077] Therefore, by combining the characteristic differences of different resource types, dynamically adjusting the parallel processing volume of target - type tasks further improves the task processing speed, system stability, and system throughput.

[0078] In the adjustment strategy, in addition to pre - configuring the execution duration threshold, in order to further improve the flexibility of the adjustment strategy, the execution duration threshold can be represented by a predetermined execution duration threshold and multiple proportionality coefficients.

[0079] Therefore, in some embodiments, determining multiple adjustment strategies associated with the resource type may include the following operations: determining a predetermined execution duration threshold associated with the resource type; generating multiple adjustment strategies according to the predetermined execution duration threshold, multiple proportionality coefficients, and the parallel processing parameters of multiple target - type tasks.

[0080] In some embodiments, the predetermined execution duration threshold can be determined according to the resource type. The multiple proportionality coefficients can be set according to actual requirements, for example: 10%, 20%, 30%, etc.

[0081] In some embodiments, the parallel processing parameter can represent the parallel processing volume. According to the rule that the larger the execution duration threshold, the smaller the corresponding parallel processing volume, multiple adjustment strategies are generated.

[0082] For example: Adjustment strategy P3: Predetermined execution duration threshold × 10%, and the corresponding parallel processing volume is 15; Adjustment strategy P4: Predetermined execution duration threshold × 20%, and the corresponding parallel processing volume is 10.

[0083] Generating adjustment strategies according to the predetermined duration threshold corresponding to the resource type, different proportionality coefficients, and the parallel processing parameters of the target - type tasks can flexibly adapt to the dynamic update of resources on the target node. For example: For any target node in the distributed storage system, when adding a RAID group, the adjustment strategy corresponding to the storage medium can be updated according to the storage medium of the newly added RAID group. Enabling the distributed system to dynamically update the adjustment strategy when perceiving resource updates, and then dynamically controlling the parallel processing volume of the target - type tasks, further improves the system stability.

[0084] In some embodiments, the multiple adjustment policies include a first adjustment policy and a second adjustment policy. According to the embodiments of the present application, determining a target adjustment policy associated with a first duration from the multiple adjustment policies may include the following operations: in response to determining that the first duration is greater than a first task execution duration threshold indicated by the first adjustment policy and less than a second task execution duration threshold indicated by the second adjustment policy, determining the first adjustment policy as the target adjustment policy; the first task execution duration threshold is less than the second task execution duration threshold; and in response to determining that the first duration is greater than or equal to the second task execution duration threshold, determining the second adjustment policy as the target adjustment policy.

[0085] For example: The adjustment policy P1 can be that when the actual duration is greater than 100 ms, the corresponding parallel processing amount is 15. The adjustment policy P2 can be that when the actual duration is greater than 150 ms, the corresponding parallel processing amount is 10.

[0086] When the first duration is 135 ms, the target adjustment policy can be determined as the adjustment policy P1. When the first duration is 160 ms, the target adjustment policy can be determined as the adjustment policy P2.

[0087] By setting the parallel processing amounts corresponding to different execution duration thresholds through gradients, the parallel processing amount of the target node for the target type of task can be dynamically adjusted according to the real-time execution status of the target type of task, further improving resource utilization.

[0088] In some embodiments, the parallel processing parameters include: the number of parallel processes. Adjusting the initial resource allocation policy according to the target adjustment policy may include the following operations: adjusting the initial number of parallel processes in the initial resource allocation policy to the target number of parallel processes in the target adjustment policy.

[0089] Figure 4A A schematic diagram showing the dynamic adjustment of the resource allocation policy according to the embodiments of the present application is shown.

[0090] As Figure 4A shown, in this embodiment 400A, the initial concurrent processing number 401 can be 5, the first duration 402 can be 35 ms, and the adjustment policy A 410A associated with the resource type is called. The following is defined in the adjustment policy A 410A: T > 40 ms, the concurrent processing number = 5; 30 < T ≤ 40 ms, the concurrent processing number = 8; 0 < T ≤ 30 ms, the concurrent processing number = 11. T represents the actual duration consumed for executing the target task.

[0091] Since the first duration 402 is 35 ms and belongs to the range of 30 < T ≤ 40 ms, it can be determined that the concurrent processing number in the target adjustment policy = 8. Therefore, the initial concurrent processing number indicated in the initial resource allocation policy can be adjusted from 5 to 8, obtaining the adjusted concurrent processing number 403.

[0092] In the adjustment strategy, the parallel processing quantity is used as the parallel processing parameter. Without performing any operations, the parallel processing parameter to be updated can be directly read from the adjustment strategy, further improving the efficiency of dynamically adjusting the parallel processing quantity.

[0093] In a distributed system, the maximum parallel processing quantity of the target type tasks of any target node is pre-configured. Therefore, during the dynamic adjustment process, it can also be adjusted based on the pre-configured maximum parallel processing quantity.

[0094] However, in actual application scenarios, the maximum parallel processing quantity of any target node is determined based on the number of nodes in the distributed system. For example, if the pre-configured maximum parallel processing quantity of the target type tasks in the distributed system is 1000 and there are 4 target nodes in the distributed system, then the maximum parallel processing quantity allocated to each target node is 250. When node updates occur in the distributed system, such as adding nodes or deleting nodes, the maximum parallel processing quantity allocated to each target node will also be updated accordingly.

[0095] Therefore, the parallel processing parameter can include: the ratio of the parallel processing quantity of the target node to the parallel processing quantity threshold of the target node, thereby improving the adaptability of the adjustment strategy to the node update scenario in the distributed system.

[0096] In some embodiments, adjusting the initial resource allocation strategy according to the target adjustment strategy may include the following operations: obtaining the target parallel processing quantity of the target node according to the parallel processing quantity threshold of the target node and the target ratio in the target adjustment strategy; adjusting the initial parallel processing quantity of the target node to the target parallel processing quantity of the target node.

[0097] In some embodiments, the parallel processing quantity threshold may represent the maximum parallel processing quantity. For example, it can be 10.

[0098] For example, the adjustment strategy P5 can be that when the actual duration is greater than 100 ms, the corresponding parallel processing quantity is 90% of the maximum parallel processing quantity. The adjustment strategy P2 can be that when the actual duration is greater than 150 ms, the corresponding parallel processing quantity is 60% of the maximum parallel processing quantity.

[0099] When the first duration is 120 ms and the matched target adjustment strategy is the adjustment strategy P5, the corresponding target ratio can be 90%, and the obtained target parallel processing quantity can be 10×90% = 9.

[0100] The initial parallel processing quantity of the target node can be adjusted to 9.

[0101] Figure 4BA schematic diagram showing dynamic adjustment of resource allocation policies according to another embodiment of the present application is shown.

[0102] As Figure 4B shown, in this embodiment 400B, the initial concurrent processing quantity 401 can be 5, the first duration 402 can be 35 ms, and the adjustment policy B 410B associated with the resource type is called. The following is defined in the adjustment policy B 410B: When T > 40 ms, the concurrent processing quantity = 30% × the maximum concurrent processing quantity; when 30 < T ≤ 40 ms, the concurrent processing quantity = 50% × the maximum concurrent processing quantity; when 0 < T ≤ 30 ms, the concurrent processing quantity = the maximum concurrent processing quantity. T represents the actual duration consumed for executing the target task. For example: The maximum concurrent processing quantity can be 16.

[0103] Since the first duration 35 ms belongs to 30 < T ≤ 40 ms, it can be determined that the concurrent processing quantity = 50% × 16 = 8. The adjusted concurrent processing quantity 403 is 8.

[0104] Figure 4C A schematic diagram showing dynamic adjustment of resource allocation policies according to yet another embodiment of the present application is shown.

[0105] As Figure 4C shown, in this embodiment 400C, the initial concurrent processing quantity 401 can be 5, the first duration 402 can be 35 ms, and the adjustment policy C 410C associated with the resource type is called. The following is defined in the adjustment policy C 410C: When T > 90%Tmax, the concurrent processing quantity = 30% × the maximum concurrent processing quantity; when 50%Tmax < T ≤ 90%Tmax, the concurrent processing quantity = 50% × the maximum concurrent processing quantity; when 0 < T ≤ 50%Tmax, the concurrent processing quantity = the maximum concurrent processing quantity. T represents the actual duration consumed for executing the target task. For example: The maximum concurrent processing quantity can be 16. Tmax represents a predetermined execution duration threshold corresponding to the resource type. For example: It can be 45 ms.

[0106] Since the first duration 35 ms belongs to 50%Tmax < T ≤ 90%Tmax, it can be determined that the concurrent processing quantity = 50% × 16 = 8. The adjusted concurrent processing quantity 403 is 8.

[0107] The execution entity of the task processing method described above is the first processor deployed on any target node in the distributed system. Since the initial resource allocation policy on any target node is calculated by the second processor for scheduling the resource configuration of multiple target nodes in the distributed system, and the adjustment policy is also pre-configured. When any target node in the distributed system dynamically adjusts the parallel processing amount of tasks of the target type, it has to adjust the parallel processing amount according to the pre-configured adjustment policy. Therefore, in order to further improve the flexible adaptation degree of the adjustment policy to the actual application scenario, the pre-configured adjustment policy can be dynamically adjusted according to the actual state of dynamically adjusting the parallel processing quantity of each node in the distributed system.

[0108] Therefore, the embodiment of the present application provides a task processing method applied to the second processor for scheduling the resource configuration of multiple target nodes deployed in the distributed system.

[0109] Figure 5 The task processing method applied to the second processor according to the embodiment of the present application is shown.

[0110] As Figure 5 shown, the method 500 may include operations S510~S520.

[0111] In operation S510, obtain multiple adjustment information of multiple target nodes within a predetermined time period.

[0112] In operation S520, based on the multiple adjustment information, adjust the parallel processing parameters indicated by the adjustment policy associated with the resource type, so that each of the multiple target nodes obtains target information according to the adjusted adjustment policy.

[0113] In the embodiment of the present application, the adjustment information may be obtained by multiple target nodes in the distributed system based on the task processing method applied to the first processor of any target node described above.

[0114] For example: The adjustment information may include but is not limited to the adjustment frequency and the adjustment direction, etc. The adjustment frequency may refer to the number of times that target nodes of the same resource type adjust the parallel processing quantity within a predetermined time period. The adjustment direction may refer to the direction in which target nodes of the same resource type adjust the parallel processing quantity within a predetermined time period, for example: increase or decrease.

[0115] In some embodiments, based on the multiple adjustment information, adjust the parallel processing parameters indicated by the adjustment policy associated with the resource type. The parallel processing parameters include but are not limited to the parallel processing quantity and the proportion of the parallel processing quantity to the parallel processing quantity threshold. So that each of the multiple target nodes obtains target information according to the adjusted adjustment policy, and the target information indicates the quantity of target type tasks processed in parallel by the target node.

[0116] For example, among the target nodes configured in a distributed system, if more than 80% of the nodes continuously reduce the number of parallel processes within a predetermined period, it indicates that the difference between the number of parallel processes corresponding to each execution duration threshold in the adjustment strategy is too small, resulting in frequent adjustments. Therefore, by expanding the difference between the number of parallel processes corresponding to each execution duration threshold, the adjustment frequency can be reduced, the matching degree of the adjustment strategy and the application scenario can be improved, and the stability of the system can be further enhanced.

[0117] The following will combine Figures 6 to 7 to detail an embodiment of adjusting the parallel processing parameters indicated by an adjustment strategy associated with a resource type based on multiple adjustment information in the above task processing method.

[0118] According to an embodiment of the present application, adjusting the parallel processing parameters indicated by an adjustment strategy associated with a resource type based on multiple adjustment information may include the following operations: determining the strategy adjustment frequency and strategy adjustment direction of multiple target nodes based on multiple adjustment information; in response to determining that the strategy adjustment frequency is greater than a predetermined frequency threshold, determining the number of first nodes with the same continuous strategy adjustment direction; and adjusting multiple parallel processing parameters indicated by the adjustment strategy according to the number of first nodes and the strategy adjustment direction.

[0119] Figure 6 FIG. shows a schematic diagram of adjusting the parallel processing parameters indicated by an adjustment strategy associated with a resource type based on multiple adjustment information according to an embodiment of the present application.

[0120] As Figure 6 shown, in Embodiment 600, first, perform operation S610 to obtain the strategy adjustment frequency and strategy adjustment direction of multiple target nodes.

[0121] In an actual application scenario, each target node can be configured with multiple types of resources. Therefore, when counting the strategy adjustment frequency and strategy adjustment direction, it is counted according to the same resource type, and the adjustment strategies corresponding to the same resource type are the same. For example: the adjustment strategy A corresponding to the SSD - type RAID group.

[0122] For example: In a distributed storage system, 4 target nodes are configured. The strategy adjustment frequency of each target node for the number of parallel processes based on adjustment strategy A is counted within a predetermined period. For example: the strategy adjustment frequency of target node S1 can be 67%, the strategy adjustment frequency of target node S2 can be 70%, the strategy adjustment frequency of target node S3 can be 40%, and the strategy adjustment frequency of target node S4 can be 75%.

[0123] Then, perform operation S620 to determine whether the strategy adjustment frequency is greater than a predetermined threshold. If not, no adjustment is made.

[0124] For example, the predetermined frequency threshold can be 60%, and it can be determined that the nodes with a policy adjustment frequency greater than the predetermined frequency threshold are target node S1, target node S2, and target node S3.

[0125] Next, for target node S1, target node S2, and target node S3, perform operation S630 to determine the number of nodes with the same policy adjustment direction.

[0126] For example, if the policy adjustment directions of target node S1, target node S2, and target node S3 are all to continuously increase the parallel processing quantity of the target type tasks, it can be determined that the number of nodes with the same policy adjustment direction is 3.

[0127] Then, perform operation S640 to determine whether the number of nodes is greater than the first predetermined quantity threshold. If not, no adjustment is made.

[0128] In some embodiments, adjusting multiple parallel processing parameters indicated by the adjustment policy according to the first node quantity and the policy adjustment direction may include the following operations: in response to determining that the first node quantity is greater than the first predetermined quantity threshold, increasing the gradient difference between the multiple parallel processing parameters indicated by the adjustment policy.

[0129] For example, the first predetermined quantity threshold can be 2. The number of nodes with the same policy adjustment direction is 3, and 3 > 2, so operation S650 can be performed to increase the gradient difference.

[0130] In some embodiments, the gradient difference can represent the difference in the concurrent processing quantities corresponding to adjacent execution duration thresholds.

[0131] For example, the gradient difference between the concurrent processing quantities corresponding to each execution duration threshold in adjustment policy A is 3. After increasing the gradient difference, the gradient difference between the concurrent processing quantities corresponding to each execution duration threshold in the obtained adjustment policy A' is 4.

[0132] To further increase the fine granularity of the gradient difference adjustment, in some embodiments, in response to determining that the first node quantity is greater than the first predetermined quantity threshold, increasing the gradient difference between the multiple parallel processing parameters indicated by the adjustment policy may include the following operations: obtaining multiple first parameter adjustment results of multiple target nodes in a predetermined time period; determining a first adjustment coefficient based on the multiple first parameter adjustment results; and increasing the gradient difference between the multiple parallel processing parameters indicated by the multiple adjustment policies based on the first adjustment coefficient.

[0133] In some embodiments, the average change rate of the multiple first parameter adjustment results can be calculated based on a statistical algorithm, and then the average change rate can be determined as the first adjustment coefficient, and the gradient difference can be increased based on the first adjustment coefficient.

[0134] For example, the first parameter adjustment result can represent a curve that continuously rises over time, and then calculate the average curvature C of this curve. Then, the absolute value of this average curvature can be used to increase the gradient difference. For example: the increased gradient difference = the initial gradient difference × the absolute value of the average curvature C.

[0135] In addition, based on the requirements of the actual scenario, other algorithms can be used to calculate the first adjustment coefficient, so that the adjusted gradient difference has a higher degree of matching with the changes in the actual parallel processing volume of each target node in the actual scenario.

[0136] By statistically analyzing the actual adjustment results of each target node, the adjustment strategy is dynamically updated, so that the adjusted gradient difference has a higher degree of matching with the changes in the actual parallel processing volume of each target node in the actual scenario, thereby reducing the frequency of adjustment of the resource allocation strategy for each target node and further improving the stability of the system.

[0137] In the embodiments of the present application, based on the policy adjustment frequency and the policy adjustment direction, adjusting multiple parallel processing parameters indicated by the adjustment policy may include the following operations: in response to determining that the policy adjustment frequency is greater than a predetermined frequency threshold, determining the number of second nodes whose policy adjustment directions fluctuate continuously; and adjusting the multiple parallel processing parameters indicated by the adjustment policy according to the number of second nodes and the policy adjustment direction.

[0138] Figure 7 Shows a schematic diagram of adjusting multiple parallel processing parameters indicated by an adjustment policy based on a policy adjustment frequency and a policy adjustment direction according to another embodiment of the present application.

[0139] As Figure 7 shown, in this embodiment 700, first, perform operation S710 to obtain the policy adjustment frequency and the policy adjustment direction of multiple target nodes.

[0140] In an actual application scenario, each target node can be configured with multiple types of resources. Therefore, when statistically analyzing the policy adjustment frequency and the policy adjustment direction, it is statistically analyzed according to the same resource type, and the adjustment policies corresponding to the same resource type are the same. For example: the adjustment policy A corresponding to the SSD type RAID group.

[0141] For example: In a distributed storage system, 4 target nodes are configured. During a predetermined time period, the policy adjustment frequency of each target node for parallel processing quantity based on the adjustment policy P is statistically analyzed. For example: the policy adjustment frequency of target node S1 can be 67%, the policy adjustment frequency of target node S2 can be 70%, the policy adjustment frequency of target node S3 can be 40%, and the policy adjustment frequency of target node S4 can be 75%.

[0142] Then, perform operation S720 to determine whether the policy adjustment frequency is greater than a predetermined threshold. If not, no adjustment is made.

[0143] For example: The predetermined frequency threshold can be 60%. It can be determined that the target nodes for which the policy adjustment frequency is greater than the predetermined frequency threshold are target node S1, target node S2, and target node S3.

[0144] Next, perform operation S730 to determine the number of nodes whose policy adjustment directions fluctuate continuously.

[0145] For example: The policy adjustment directions of target node S1, target node S2, and target node S3 all fluctuate continuously. It can be determined that the number of nodes is 3.

[0146] Then, perform operation S740 to determine whether the number of nodes is greater than a second predetermined quantity threshold. If not, no adjustment is made.

[0147] In some embodiments, adjusting a plurality of parallel processing parameters indicated by an adjustment policy according to the second node quantity and the policy adjustment direction may include the following operations: In response to determining that the second node quantity is greater than the second predetermined quantity threshold, reducing the gradient difference between the plurality of parallel processing parameters indicated by the plurality of adjustment policies.

[0148] For example: The second predetermined quantity threshold can be 2, and the number of nodes whose policy adjustment directions fluctuate continuously is 3, 3 > 2. Therefore, perform operation S750 to reduce the gradient difference.

[0149] In some embodiments, the gradient difference may characterize the difference in the number of concurrent processes corresponding to adjacent execution duration thresholds.

[0150] For example: The gradient difference between the number of concurrent processes corresponding to each execution duration threshold in adjustment policy A is 3. After reducing the gradient difference, the gradient difference between the number of concurrent processes corresponding to each execution duration threshold in the obtained adjustment policy A" is 2.

[0151] In some embodiments, in response to determining that the second node quantity is greater than the second predetermined quantity threshold, reducing the difference between the parallel processing parameters indicated by the plurality of adjustment policies may include the following operations: Obtain a plurality of second parameter adjustment results of a plurality of target nodes in a predetermined period; determine a second adjustment coefficient based on the plurality of second parameter adjustment results; and reduce the gradient difference between the parallel processing parameters indicated by the plurality of adjustment policies based on the second adjustment coefficient.

[0152] In some embodiments, multiple second parameter adjustment results can be arranged in the chronological order of the adjustment results to generate a time series of the parallel processing quantity. Then, a time series neural network, such as an LSTM (Long Short-Term Memory) network, can be used to process the time series of the parallel processing quantity to generate a change rate of the parallel processing quantity. And this change rate can be used as the second adjustment coefficient to reduce the gradient difference based on the second adjustment coefficient.

[0153] By statistically analyzing the actual adjustment results of each target node, the adjustment strategy is dynamically updated, so that the adjusted gradient difference has a higher matching degree with the change of the actual parallel processing quantity of each target node in the actual scenario, thereby reducing the frequency of adjustment of the resource allocation strategy of each target node and further improving the stability of the system.

[0154] In the embodiments of the present application, the above method may further include the following operations: obtaining the resource allocation status of multiple target nodes in a historical period and the historical execution duration of target type tasks; generating the resource requirements of each of the multiple target nodes in a first period based on the historical resource allocation status and the historical execution duration; and generating an initial resource allocation strategy for each of the multiple target nodes based on the resource requirements of each of the multiple target nodes.

[0155] In some embodiments, the resource allocation status may represent the historical parallel processing quantity for target type tasks.

[0156] In some embodiments, generating the resource requirements of a target node in a first period based on the historical resource allocation status and the historical execution duration may include the following operations: extracting the historical resource allocation status features and the historical execution duration features; and inputting the historical resource allocation status features and the historical execution duration features into a trained time series neural network to generate the resource requirements.

[0157] In some embodiments, the resource requirements may represent the requirements for the parallel processing quantity of each target node for target type tasks in a future period.

[0158] In some embodiments, the time series neural network may be an LSTM network. The sample concurrent processing quantity and the sample execution duration of the target type tasks in the t-th period can be used as inputs and input into the LSTM network to output a predicted value or a predicted sequence of the concurrent processing quantity of the target type tasks in the (t + 1)-th period. t may be an integer greater than 1. Then, any loss function applicable to deep learning algorithms, such as a cross-entropy loss function, can be used to calculate the loss value based on the predicted value or the predicted sequence and the sample label. The sample label may be the true value or the true sequence of the concurrent processing quantity of the target type tasks in the (t + 1)-th period. Then, based on the loss value, the model parameters of the LSTM network are adjusted until the loss value converges to obtain a trained time series neural network.

[0159] Predict the change in the concurrent processing volume of each target node using the trained time-series neural network, so that the initial resource allocation strategy matches the actual load status of each target node.

[0160] In some embodiments, generating an initial resource allocation strategy based on resource requirements may include the following operations: determining the resource allocation ratios of multiple target nodes based on the resource requirements of the multiple target nodes; and allocating predetermined resources according to the resource allocation ratios to generate an initial resource allocation strategy.

[0161] For example: The parallel processing quantity threshold for target type tasks in a distributed system is 100, and the resource allocation ratios of each target node calculated according to the actual resource requirements of each target node are 1:2:4:3. Therefore, the initial parallel processing quantities for target type tasks allocated to each target node in the initial resource allocation strategy are: 10, 20, 40, 30.

[0162] Allocating resources according to the actual resource requirements of each target node further reduces the probability of local overload.

[0163] In an actual application scenario, when each target node executes the target type tasks to be processed according to its respective concurrent processing quantity, partial node failures may occur, resulting in the tasks not completed on the faulty nodes being unable to continue execution.

[0164] In view of this, the task processing method provided by the embodiments of the present application may further include the following operations: in response to determining that any one of the multiple target nodes is operating abnormally, obtaining the task execution status of the abnormal node; and scheduling the tasks not completed on the abnormal node to the idle nodes among the multiple target nodes for continued execution.

[0165] For example: In a distributed system, target node S1, target node S2, and target node S3 are configured. The operating status of each node can be detected through heartbeat detection or status polling. When it is determined that target node S2 fails, the tasks not completed in target node S2 can be scheduled to target node S1 or target node S3. During the task scheduling process, the tasks not completed are preferentially scheduled to the idle nodes. Thus, the continuity of task execution in the entire distributed system is ensured.

[0166] Figure 8 Shows a schematic diagram of a task processing device applied to a distributed system according to an embodiment of the present application.

[0167] As Figure 8 shown, the task processing device 800 may include a first processor and a second processor.

[0168] The first processor can be configured in the Local Scheduler (LS) of each target node in a distributed system. In this distributed system, n target nodes can be configured. Therefore, the corresponding local schedulers respectively include: LS1 821, LS2 822, …, LSn 82n.

[0169] The first processor is configured to receive an initial resource allocation policy from the second processor; and obtain a first duration consumed by the target node for executing multiple tasks of a target type in the current period and the resource types configured on the target node; wherein the resources configured on the target node for executing tasks of the target type are determined according to the initial resource allocation policy; the initial resource allocation policy indicates the quantity information of the target type tasks processed in parallel by the target node; adjust the initial resource allocation policy based on the first duration and the resource types; and process multiple second tasks of the target type in parallel in a subsequent period according to the target quantity information indicated by the adjusted resource allocation policy.

[0170] The second processor can be configured in the GRM810 (Global Resource Manager, global resource scheduling server) in the distributed system.

[0171] The second processor is configured to send the initial resource allocation policy to the first processor; and obtain multiple adjustment information for adjusting the initial resource allocation policy of multiple target nodes within a predetermined period; and adjust the parallel processing parameters indicated by the adjustment policy based on the multiple adjustment information.

[0172] In some embodiments, first, the second processor sends the initial resource allocation policy to the first processor. Then, the first processor receives the initial resource allocation policy from the second processor; and obtains the first duration consumed by the target node for executing multiple tasks of the target type in the current period and the resource types configured on the target node.

[0173] Next, the first processor adjusts the initial resource allocation policy based on the first duration and the resource types; and processes multiple second tasks of the target type in parallel in a subsequent period according to the target quantity information indicated by the adjusted resource allocation policy.

[0174] Then, obtain multiple adjustment information for adjusting the initial resource allocation policy of multiple target nodes within a predetermined period; and adjust the parallel processing parameters indicated by the adjustment policy based on the multiple adjustment information. So that the second processors distributed on each target node can dynamically adjust the parallel processing quantity of the target type tasks according to the updated adjustment policy.

[0175] In the embodiments of the present application, the first processor in each local scheduling server configured at the target node may adjust the parallel processing quantity of the target type tasks based on the adjustment policy, and the second processor configured in the global resource scheduling server may dynamically update the adjustment policy according to the adjustment results at each target node, realizing the dynamic optimization of task scheduling and improving the overall performance stability of the distributed system.

[0176] In some embodiments, the first processor is further configured to determine a plurality of adjustment policies associated with the resource type; the adjustment policy indicates the association relationship between the execution duration threshold and the parallel processing parameters of the target type tasks; determine the target adjustment policy associated with the first duration from the plurality of adjustment policies; and adjust the initial resource allocation policy according to the target adjustment policy.

[0177] By combining the characteristic differences of different resource types and dynamically adjusting the parallel processing quantity of the target type tasks, the task processing speed, system stability, and system throughput are further improved.

[0178] In some embodiments, the second processor is further configured to determine the policy adjustment frequency and policy adjustment direction of a plurality of target nodes based on a plurality of adjustment information; in response to determining that the policy adjustment frequency is greater than a predetermined frequency threshold, determine the number of first nodes with the same policy adjustment direction continuously; and adjust the parallel processing parameters indicated by the adjustment policy according to the number of first nodes and the policy adjustment direction.

[0179] In some embodiments, the second processor is further configured to, in response to determining that the policy adjustment frequency is greater than a predetermined frequency threshold, determine the number of second nodes with continuously fluctuating policy adjustment directions; and adjust a plurality of parallel processing parameters indicated by the adjustment policy according to the number of second nodes and the policy adjustment direction.

[0180] By statistically analyzing the actual adjustment results of each target node and dynamically updating the adjustment policy, the matching degree between the adjusted gradient difference and the change in the actual parallel processing quantity of each target node in the actual scenario is higher, thereby reducing the frequency of resource allocation policy adjustment for each target node and further improving the system's...

[0181] In some embodiments, in order to further increase system stability, in the face of the situation where some nodes in the distributed system may fail, the task processing device in the embodiments of the present application may further include: ARE830 (Abnormal Recovery Engine), which is used to detect the running status of each target node and feedback it to GRM810. So that when any target node fails, GRM810 schedules the tasks not completed by the failed node to other nodes in the distributed system for continued execution.

[0182] It should be noted that the embodiments of the present application are also applicable to the virtual machine resource allocation scenario in a cloud computing environment. In a cloud computing environment, each target node can be a virtual machine in the cloud computing environment. The first processor can adjust the number of parallel processes of non-business tasks in the virtual machine according to the feedback of the real-time resource usage status of the virtual machine. The second processor can send an initial resource allocation policy to each virtual machine. The embodiments of the present application can also be applicable to other distributed systems, which will not be elaborated here.

[0183] Figure 9 A block diagram of an electronic device suitable for implementing a task processing method according to an embodiment of the present application is shown.

[0184] As Figure 9 shown, the electronic device 900 according to an embodiment of the present application includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage section 908 into a random access memory (RAM) 903. The processor 901 can include, for example, a general microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as an application specific integrated circuit (ASIC)), etc. The processor 901 can also include on-board memory for caching purposes. The processor 901 can include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present application.

[0185] In the RAM 903, various programs and data required for the operation of the electronic device 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. The processor 901 performs various operations of the method flow according to an embodiment of the present application by executing the programs in the ROM 902 and / or the RAM 903. It should be noted that the program can also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 can also perform various operations of the method flow according to an embodiment of the present application by executing the programs stored in the one or more memories.

[0186] According to an embodiment of the present application, the electronic device 900 may further include an input / output (I / O) interface 905, and the input / output (I / O) interface 905 is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the input / output (I / O) interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. The drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0187] The present application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist separately without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiments of the present application is implemented.

[0188] According to an embodiment of the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, the computer-readable storage medium may include the above-described ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903.

[0189] An embodiment of the present application also includes a computer program product, which includes a computer program, and the computer program includes program codes for executing the method shown in the flowchart. When the computer program product runs in a computer system, the program codes are used to cause the computer system to implement the task processing method provided by the embodiments of the present application.

[0190] When the computer program is executed by the processor 901, the above functions defined in the system / apparatus of the embodiments of the present application are executed. According to the embodiments of the present application, the systems, apparatuses, modules, units, etc. described above can be implemented by computer program modules.

[0191] In one embodiment, the computer program can rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program can also be transmitted and distributed in the form of signals on a network medium, and be downloaded and installed through the communication part 909, and / or be installed from the removable medium 911. The program code included in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0192] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or be installed from the removable medium 911. When the computer program is executed by the processor 901, the above functions defined in the system of the embodiments of the present application are executed. According to the embodiments of the present application, the systems, devices, apparatuses, modules, units, etc. described above can be implemented by computer program modules.

[0193] According to the embodiments of the present application, the program code for executing the computer program provided by the embodiments of the present application can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level procedures and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include but are not limited to, such as Java, C++, python, the "C" language, or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, by using an Internet service provider to connect through the Internet).

[0194] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code, and the above-mentioned module, segment of a program, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, as well as the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0195] Those skilled in the art can understand that the features described in various embodiments of the present application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in various embodiments of the present application can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present application.

[0196] The embodiments of the present application have been described above. However, these embodiments are only for illustrative purposes and not for limiting the scope of the present application. Although the embodiments have been described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Without departing from the scope of the present application, those skilled in the art can make various substitutions and modifications, and all such substitutions and modifications should fall within the scope of the present application.

Claims

1. A task processing method, characterized in that, A first processor applied to any target node deployed in a distributed system; the method includes: Obtain the first duration consumed by the target node to execute multiple first tasks of a target type during the current period and the resource types configured on the target node; wherein, the resources configured on the target node for executing tasks of the target type are determined according to an initial resource allocation policy; the initial resource allocation policy indicates the initial quantity information of the target type tasks processed in parallel by the target node; and Based on the first duration and the resource types, adjust the initial resource allocation policy; The target node processes multiple second tasks of the target type in parallel during the subsequent period according to the target quantity information indicated by the adjusted resource allocation policy.

2. The method according to claim 1, wherein The adjusting the initial resource allocation policy based on the first duration and the resource types includes: Determine multiple adjustment policies associated with the resource types; the adjustment policies indicate the association relationship between the execution duration threshold and the parallel processing parameters of the target type tasks; Determine a target adjustment policy associated with the first duration from the multiple adjustment policies; Adjust the initial resource allocation policy according to the target adjustment policy.

3. The method according to claim 2, characterized in that, The execution duration threshold includes: a predetermined execution duration threshold and multiple proportionality coefficients; the determining the multiple adjustment policies associated with the resource types includes: Determine the predetermined execution duration threshold associated with the resource types; Generate the multiple adjustment policies according to the predetermined execution duration threshold, the multiple proportionality coefficients and the parallel processing parameters of the multiple target type tasks.

4. The method according to claim 2 or 3, characterized in that, The multiple adjustment policies include a first adjustment policy and a second adjustment policy; The determining the target adjustment policy associated with the first duration from the multiple adjustment policies includes: In response to determining that the first duration is greater than the first task execution duration threshold indicated by the first adjustment policy and less than the second task execution duration threshold indicated by the second adjustment policy, determine the first adjustment policy as the target adjustment policy; the first task execution duration threshold is less than the second task execution duration threshold; And In response to determining that the first duration is greater than or equal to the second task execution duration threshold, determine the second adjustment policy as the target adjustment policy.

5. The method according to claim 2, characterized in that The parallel processing parameters include: the number of parallel processes; The adjusting the initial resource allocation policy according to the target adjustment policy includes: Adjust the initial number of parallel processes in the initial resource allocation policy to the target number of parallel processes in the target adjustment policy.

6. The method according to claim 2, wherein The parallel processing parameters include: the ratio of the number of parallel processes of the target node to the threshold of the number of parallel processes of the target node; The adjusting the initial resource allocation policy according to the target adjustment policy includes: For any target node, obtain the target number of parallel processes of the target node according to the threshold of the number of parallel processes of the target node and the target ratio in the target adjustment policy; Adjust the initial number of parallel processes of the target node to the target number of parallel processes of the target node.

7. A task processing method, applied to a second processor deployed in a distributed system for scheduling resource configurations of multiple target nodes; the method includes: Obtaining multiple adjustment information of multiple target nodes within a predetermined time period; wherein, the multiple adjustment information is obtained based on the method according to any one of claims 1 to 6; and Based on the multiple adjustment information, adjusting the parallel processing parameters indicated by an adjustment strategy associated with a resource type, so that each of the multiple target nodes obtains target information according to the adjusted adjustment strategy, and the target information indicates the number of target type tasks processed in parallel by the target node.

8. The method according to claim 7, wherein The adjusting the parallel processing parameters indicated by the adjustment strategy associated with the resource type based on the multiple adjustment information includes: Based on the multiple adjustment information, determining the policy adjustment frequency and the policy adjustment direction of the multiple target nodes; In response to determining that the policy adjustment frequency is greater than a predetermined frequency threshold, determining the number of first nodes with the same continuous policy adjustment direction; and Adjusting the multiple parallel processing parameters indicated by the adjustment strategy according to the number of first nodes and the policy adjustment direction.

9. The method according to claim 8, characterized in that The adjusting the multiple parallel processing parameters indicated by the adjustment strategy according to the number of first nodes and the policy adjustment direction includes: In response to determining that the number of first nodes is greater than a first predetermined number threshold, increasing the gradient difference between the multiple parallel processing parameters indicated by the adjustment strategy.

10. The method according to claim 9, wherein The increasing the gradient difference between the multiple parallel processing parameters indicated by the adjustment strategy in response to determining that the number of first nodes is greater than a first predetermined number threshold includes: Obtaining multiple first parameter adjustment results of the multiple target nodes within a predetermined time period; Based on the multiple first parameter adjustment results, determining a first adjustment coefficient; and Based on the first adjustment coefficient, increasing the gradient difference between the multiple parallel processing parameters indicated by the multiple adjustment strategies.

11. The method according to claim 8, wherein The adjusting the multiple parallel processing parameters indicated by the adjustment strategy based on the policy adjustment frequency and the policy adjustment direction includes: In response to determining that the policy adjustment frequency is greater than a predetermined frequency threshold, determining the number of second nodes with continuous fluctuations in the policy adjustment direction; and Adjusting the multiple parallel processing parameters indicated by the adjustment strategy according to the number of second nodes and the policy adjustment direction.

12. The method according to claim 11, wherein The adjusting the multiple parallel processing parameters indicated by the adjustment strategy according to the number of second nodes and the policy adjustment direction includes: In response to determining that the number of second nodes is greater than a second predetermined number threshold, decreasing the gradient difference between the multiple parallel processing parameters indicated by the multiple adjustment strategies.

13. The method according to claim 11, wherein The decreasing the gradient difference between the parallel processing parameters indicated by the multiple adjustment strategies in response to determining that the number of second nodes is greater than a second predetermined number threshold includes: Obtaining multiple second parameter adjustment results of the multiple target nodes within a predetermined time period; Based on the multiple second parameter adjustment results, determining a second adjustment coefficient; and Based on the second adjustment coefficient, decreasing the gradient difference between the multiple parallel processing parameters indicated by the multiple adjustment strategies.

14. The method according to claim 7, wherein The method further includes: Obtain the resource allocation status of multiple target nodes in the historical period and the historical execution duration of target type tasks; Based on the historical resource allocation status and the historical execution duration, generate the resource requirements of each of the multiple target nodes in the first period; and Based on the resource requirements of each of the multiple target nodes, generate the initial resource allocation strategy for each of the multiple target nodes.

15. The method according to claim 14, wherein The generating the resource requirements of the target node in the first period based on the historical resource allocation status and the historical execution duration includes: Extract the historical resource allocation status features and historical execution duration features; and Input the historical resource allocation status features and historical execution duration features into the trained time series neural network to generate the resource requirements.

16. The method according to claim 14, characterized in that The generating the initial resource allocation strategy based on the resource requirements includes: Based on the resource requirements of multiple target nodes, determine the resource allocation ratio of multiple target nodes; and Allocate the predetermined resources according to the resource allocation ratio to generate the initial resource allocation strategy.

17. The method according to claim 16, wherein The method further includes: In response to determining that any one of the multiple target nodes is operating abnormally, obtain the task execution status of the abnormal node; and Schedule the tasks not completed by the abnormal node to the idle nodes among the multiple target nodes for continued execution.

18. A task processing device, applied to a distributed system, is characterized in that The device includes: A first processor, configured to receive the initial resource allocation strategy from a second processor; and obtain the first duration consumed by the target node to execute multiple tasks of the target type in the current period and the resource types configured on the target node; wherein, the resources configured on the target node for executing tasks of the target type are determined according to the initial resource allocation strategy; the initial resource allocation strategy indicates the quantity information of the target type tasks processed in parallel by the target node; adjust the initial resource allocation strategy based on the first duration and the resource types; and the target node processes multiple second tasks of the target type in parallel in the subsequent period according to the target quantity information indicated by the adjusted resource allocation strategy; A second processor, configured to send the initial resource allocation strategy to the first processor; and obtain multiple adjustment information of multiple target nodes within a predetermined period; and adjust the parallel processing parameters indicated by the adjustment strategy based on the multiple adjustment information; wherein, the adjustment information is obtained according to the method of any one of claims 1 to 6.

19. The device according to claim 18, characterized in that, The first processor is further configured to: Determine multiple adjustment strategies associated with the resource types; the adjustment strategies indicate the association relationship between the execution duration threshold and the parallel processing parameters of the target type tasks; Determine the target adjustment strategy associated with the first duration from the multiple adjustment strategies; Adjust the initial resource allocation strategy according to the target adjustment strategy.

20. The device according to claim 18, characterized in that The second processor is further configured to: Based on the multiple adjustment information, determine the strategy adjustment frequency and the strategy adjustment direction of multiple target nodes; In response to determining that the strategy adjustment frequency is greater than a predetermined frequency threshold, determine the number of first nodes with the same strategy adjustment direction continuously; And Adjust the parallel processing parameters indicated by the adjustment policy according to the first node quantity and the policy adjustment direction.

21. The device according to claim 18, characterized in that, The second processor is further configured to: In response to determining that the policy adjustment frequency is greater than a predetermined frequency threshold, determine a second node quantity with continuously fluctuating policy adjustment directions; and Adjust a plurality of parallel processing parameters indicated by the adjustment policy according to the second node quantity and the policy adjustment direction.

22. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

Images