Task processing methods, devices and storage media

Abstract

This application provides a task processing method applicable to the field of computer technology. The task processing method includes: obtaining a first duration consumed by a target node in executing multiple first tasks of a target type during the current time period and the resource types configured on the target node; wherein the resources configured on the target node for executing the target type tasks are determined according to an initial resource allocation strategy; the initial resource allocation strategy indicates the initial number of target type tasks that the target node will process in parallel; adjusting the initial resource allocation strategy based on the first duration and resource types; and the target node processing multiple second tasks of the target type in parallel in subsequent time periods according to the target number information indicated by the adjusted resource allocation strategy. This application also provides a task processing device and a storage medium.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to the field of resource management and task scheduling technology, and more specifically to a task processing method, device, and storage medium. Background Technology

[0002] With the increasing volume of data processing in distributed systems, especially in high-concurrency scenarios involving business tasks, non-business tasks, such as data destruction tasks, compete with business tasks for bandwidth, CPU, and memory resources, which can easily lead to delays or interruptions in normal business processing. Summary of the Invention

[0003] In view of the above problems, this application provides a task processing method, apparatus and storage medium.

[0004] The first aspect of this application provides a task processing method applied to a first processor deployed on any target node in a distributed system, comprising: obtaining a first duration consumed by the target node in executing multiple first tasks of a target type in the current time period and the resource types configured on the target node; wherein the resources configured on the target node for executing the target type tasks are determined according to an initial resource allocation strategy; the initial resource allocation strategy indicates the initial number of target type tasks processed in parallel by the target node; adjusting the initial resource allocation strategy based on the first duration and resource types; and the target node processing multiple second tasks of the target type in parallel in subsequent time periods according to the target number information indicated by the adjusted resource allocation strategy.

[0005] A second aspect of this application provides a task processing method for a second processor deployed in a distributed system for scheduling resource configurations of multiple target nodes, comprising: acquiring multiple adjustment information of multiple target nodes within a predetermined time period; wherein the multiple adjustment information is obtained based on the task processing method described above applied to a first processor; and adjusting the parallel processing parameters indicated by an adjustment strategy associated with resource type based on the multiple adjustment information, so that each of the multiple target nodes obtains target information according to the adjusted adjustment strategy, wherein the target information indicates the number of target type tasks that the target nodes process in parallel.

[0006] A third aspect of this application provides a task processing device, comprising: a first processor and a second processor.

[0007] A first processor is configured to receive an initial resource allocation strategy from a second processor; and acquire a first duration consumed by the target node in executing multiple tasks of the target type during the current time period and the resource types configured on the target node; wherein the resources configured on the target node for executing the target type tasks are determined according to the initial resource allocation strategy; the initial resource allocation strategy indicates the number of target type tasks that the target node processes in parallel; adjust the initial resource allocation strategy based on the first duration and resource types; and the target node processes multiple second tasks of the target type in parallel in subsequent time periods according to the target number information indicated by the adjusted resource allocation strategy.

[0008] The second processor is used to send an initial resource allocation strategy to the first processor; and to obtain multiple adjustment information for adjusting the initial resource allocation strategy for multiple target nodes within a predetermined time period; and to adjust the parallel processing parameters indicated by the adjustment strategy based on the multiple adjustment information.

[0009] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0010] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0011] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0012] Figure 1 An application scenario diagram of the task processing method according to an embodiment of this application is shown;

[0013] Figure 2 A flowchart of a task processing method applied to a first processor according to an embodiment of this application is shown;

[0014] Figure 3 A flowchart illustrating the time consumed by statistical target type tasks according to embodiments of this application is shown;

[0015] Figure 4A A schematic diagram illustrating the dynamic adjustment of resource allocation strategies according to embodiments of this application is shown;

[0016] Figure 4B A schematic diagram illustrating the dynamic adjustment of resource allocation strategies according to another embodiment of this application is shown;

[0017] Figure 4CA schematic diagram illustrating the dynamic adjustment of resource allocation strategies according to yet another embodiment of this application is shown;

[0018] Figure 5 This invention illustrates a task processing method applied to a second processor according to an embodiment of the present application;

[0019] Figure 6 This illustration shows a schematic diagram of how, according to an embodiment of this application, the parallel processing parameters are adjusted based on an adjustment strategy associated with a resource type, according to multiple adjustment information.

[0020] Figure 7 A schematic diagram is shown illustrating the adjustment of multiple parallel processing parameters indicated by the adjustment strategy based on the strategy adjustment frequency and the strategy adjustment direction, according to another embodiment of this application.

[0021] Figure 8 A schematic diagram of a task processing device according to an embodiment of this application is shown;

[0022] Figure 9 A block diagram of an electronic device suitable for implementing a task processing method according to an embodiment of this application is shown. Detailed Implementation

[0023] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0027] A distributed system is a system consisting of a group of computer nodes that communicate over a network and coordinate their work to accomplish a common task. Examples include distributed storage systems and distributed virtual machine systems in a cloud computing environment.

[0028] The number of tasks processed by distributed storage systems and cloud platforms is growing exponentially. In high-concurrency data scenarios, there are business tasks and non-business tasks, such as data destruction tasks competing for bandwidth, CPU, and memory resources, which can lead to interruptions or delays in business tasks.

[0029] Taking data destruction tasks in a distributed storage system as an example, when faced with fluctuations in the processing volume of business tasks, business tasks and non-business tasks will compete for server bandwidth, CPU, or memory resources. Related examples typically employ static quota methods, token bucket algorithms, feedback control methods, or priority scheduling methods for resource allocation.

[0030] However, the examples provided are for non-business tasks, such as data destruction tasks, and the resource allocation methods provided have certain limitations.

[0031] For example, the fixed quotas configured using the static quota method cannot respond to load fluctuations. During periods of high load, a quota of 5 tasks per node can easily lead to task backlog, increasing the average waiting time from minutes to hours and slowing down data processing efficiency. Conversely, during periods of low business task volume, fixed quotas result in idle computing and storage resources, making it impossible to flexibly allocate them to other tasks, leading to wasted hardware investment.

[0032] For example, during large-scale database updates, the token bucket algorithm may experience a surge in token destruction tasks that far exceed the token generation rate, causing task latency to jump from milliseconds to seconds, impacting business performance. HDD and SSD RAID groups exhibit significant performance differences, but the token allocation strategy is not specifically optimized for them. For instance, SSD RAID groups can handle higher concurrency due to their high-speed read / write capabilities, while HDD RAID groups are prone to performance bottlenecks due to insufficient tokens.

[0033] For example, feedback control adjusts task rates by dynamically monitoring system load (such as CPU utilization). However, HDD RAID 5 experiences a sharp performance drop when handling random read / write operations, while SSD RAID 0 can handle it efficiently. The feedback mechanism, however, doesn't adjust according to media type, leading to overload of HDD groups and idle resources in SSD groups. In cloud storage scenarios, when user requests surge, the system needs several seconds to collect metrics and adjust rates, potentially causing a sharp performance drop and impacting user experience.

[0034] For example, priority scheduling requires configuring priorities for different application scenarios, which has poor versatility. Furthermore, in some special scenarios, strict deadlines are set for data destruction. Prioritizing business tasks may result in the failure to destroy specific data on time, leading to data security risks.

[0035] In view of this, embodiments of this application provide a task processing method, including: obtaining a first duration consumed by a target node in executing multiple first tasks of a target type during the current time period and the resource types configured on the target node; wherein the resources configured on the target node for executing target type tasks are determined according to an initial resource allocation strategy; the initial resource allocation strategy indicates the initial number of target type tasks to be processed in parallel by the target node; adjusting the initial resource allocation strategy based on the first duration and resource types; and processing multiple second tasks of the target type in parallel in subsequent time periods according to the target number information indicated by the adjusted resource allocation strategy.

[0036] Figure 1 An application scenario diagram of the task processing method according to an embodiment of this application is shown.

[0037] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 1011, a second terminal device 1012, a third terminal device 1013, and a first web server 1021, a second web server 1022, and a third web server 1023 that interact with the first terminal device 1011, the second terminal device 1012, and the third terminal device 1013 respectively.

[0038] Users can use the first terminal device 1011, the second terminal device 1012, and the third terminal device 1013 to interact with the first web server 1021, the second web server 1022, and the third web server 1023 respectively via the network to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 1011, the second terminal device 1012, and the third terminal device 1013, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only). Information received and processed by the first web server 1021, the second web server 1022, and the third web server 1023 can be stored in the distributed storage system 103. Multiple storage servers can be configured in the distributed storage system to store information from the first web server 1021, the second web server 1022, and the third web server 1023, or to send information to be retrieved from the first web server 1021, the second web server 1022, and the third web server 1023.

[0039] The first terminal device 1011, the second terminal device 1012, and the third terminal device 1013 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0040] The first web server 1021, the second web server 1022, and the third web server 1023 can be servers that provide various services, such as a backend management server that supports websites browsed by users using the first terminal device 1011, the second terminal device 1012, and the third terminal device 1013 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0041] It should be noted that the task processing method provided in this application embodiment can generally be executed by the storage server in the distributed storage system 103.

[0042] It should be understood that Figure 1 The number of terminal devices, web servers, and storage servers in the distributed storage system shown is merely illustrative. Depending on implementation needs, any number of terminal devices, web servers, and storage servers in the distributed storage system can be included.

[0043] The following will be based on Figure 1 The described scene, through Figures 2-8 The task processing method of the embodiment will be described in detail.

[0044] Figure 2 A flowchart of a task processing method applied to a first processor according to an embodiment of this application is shown.

[0045] like Figure 2 As shown, the task processing method 200 of this embodiment includes operations S210 to S230.

[0046] In operation S210, obtain the first duration consumed by the target node in executing multiple first tasks of the target type during the current time period and the resource types configured on the target node.

[0047] In operation S220, the initial resource allocation strategy is adjusted based on the first duration and resource type.

[0048] In operation S230, the target node processes multiple second tasks of the target type in parallel in subsequent time periods according to the target quantity information indicated by the adjusted resource allocation strategy.

[0049] In this embodiment of the application, for a distributed storage system, the target node can be any storage server in the distributed storage system. The resource type configured on the target node can be a storage medium configured on the storage server, such as: HDD (Hard Disk Drive), SSD (Solid State Drive), NVMe (Non-Volatile Memory Express), SCM (Storage Class Memory), etc.

[0050] In some embodiments, the resources configured on the target node for executing target-type tasks are determined according to an initial resource allocation strategy. The initial resource allocation strategy indicates the initial number of target-type tasks that the target node will process in parallel.

[0051] In this embodiment, the target type task can be a non-business task, such as a data destruction task. The initial quantity information can be calculated based on the predetermined maximum parallel processing capacity and number of nodes for the data destruction task.

[0052] For example, if the parallel processing volume of the maximum data destruction task is 1000 and the number of target nodes configured on the distributed storage system is 4, the initial quantity information of each target node can be determined to be 250.

[0053] In some embodiments, business tasks and target type tasks compete for resources during execution, and the processing volume of business tasks fluctuates irregularly due to the influence of the user end. Therefore, by obtaining the first duration consumed by the target node in executing multiple first tasks of the target type in the current time period, the parallel processing volume can be dynamically allocated to the target type tasks in a refined manner. This does not affect the allocation of bandwidth and computing power resources such as input and output of the target node to the business tasks, further reducing performance jitter caused by resource contention in high-concurrency scenarios of business tasks and improving system stability.

[0054] In this embodiment of the application, the first duration may be the average duration consumed by a predetermined number of target type tasks, and the first duration is obtained by concurrently processing data destruction tasks according to the initial number of concurrent processing tasks.

[0055] Figure 3 A flowchart illustrating the time consumed by a statistical target type task according to an embodiment of this application is shown.

[0056] like Figure 3 As shown, the target type tasks to be processed by the target node can be executed in the order of arrival at the target node, or in the order of priority of the target type tasks. First, the task start time 301 can be recorded when the target type task starts to execute, and the task end time 302 can be recorded when the target type task ends.

[0057] Then, execute operation S310 to calculate the completion time. And execute operation S320 to record the number of times the task was completed.

[0058] Next, operation S330 is executed to determine whether the number of times the task has been completed is equal to α. α can represent a predetermined number of times, for example, 1000 times.

[0059] If yes, then execute operation S340 to calculate the average duration. For example, the average duration can be obtained by dividing the actual duration of 1000 tasks by 1000. If no, then execute operation S350 to execute the next task and calculate the completion time of the next task.

[0060] In some embodiments, multiple RAID (Redundant Arrays of Independent Disks) groups are configured on the target node. The initial number of each RAID group can be further calculated so that the target node can schedule target-type tasks based on the initial number of each RAID group.

[0061] In some embodiments, the number of RAID groups configured on the target node is also dynamically updated, and the parallel processing capacity of each RAID group can be dynamically updated according to the actual number of RAID groups.

[0062] Different storage media have different characteristics. For example, HDDs have high throughput but random read / write latency in the hundreds of milliseconds, making them suitable for sequential access scenarios. SSDs or NVMe have lower read / write latency, only in the 30-millisecond range, but they suffer from write amplification, resulting in actual write volumes far exceeding user needs. SCMs can reduce random read / write latency to the nanosecond level, but their high cost makes large-scale deployment difficult.

[0063] Therefore, when dynamically adjusting the number of concurrent processing tasks of a target type, it is necessary to specify the storage medium type in order to further improve resource utilization.

[0064] For example, two RAID groups can be configured on the target node: one RAID group for SSDs and the other for HDDs. Due to the difference in storage media performance, the random read / write latency of SSDs is typically 30ms, while that of HDDs is typically 100ms.

[0065] Therefore, the average time consumed by performing data destruction tasks on RAID groups of the same type of storage media can be used to further improve the accuracy of the dynamic adjustment strategy results.

[0066] In some embodiments, the amount of parallel processing indicated in the initial resource allocation strategy can be adjusted based on the degree and direction of deviation of the first duration from the predetermined delay of the RAID group. The direction of adjustment of the amount of parallel processing is opposite to the direction of deviation, and the degree of adjustment can be the same as the degree of deviation. The predetermined delay can be determined based on the performance of the RAID group.

[0067] For example, for a RAID group with HDDs as the storage medium, the initial resource allocation strategy indicates a parallel processing capacity of 15 for data destruction tasks, with an initial duration of 120ms. The predetermined latency of the HDDs can be 100ms. It can be determined that the current initial duration deviates from the predetermined latency of the RAID group by 12%. Accordingly, the parallel processing capacity can be reduced by 12%, adjusted to 13. This means that in the next time period, for a RAID group with HDDs as the storage medium, a maximum of 13 data destruction tasks can be processed in parallel.

[0068] In this embodiment, based on the actual time consumed by the target node in processing the target type task in the current time period, and combined with the performance characteristics of the resource type configured by the target node, the parallel processing volume of the target type task in the next time period is dynamically adjusted. This solves at least the problem of system performance instability and business interruption or delay caused by the target node competing for resources with business tasks when processing the target type task. It achieves the technical effect of optimizing concurrent task allocation and system throughput efficiency based on the performance characteristics of resource type.

[0069] According to an embodiment of this application, adjusting the initial resource allocation strategy based on a first duration and a resource type may include the following operations: determining multiple adjustment strategies associated with the resource type; determining a target adjustment strategy associated with the first duration from the multiple adjustment strategies; and adjusting the initial resource allocation strategy according to the target adjustment strategy.

[0070] In some embodiments, the adjustment strategy indicates the correlation between an execution duration threshold and the parallel processing parameters of the target type task. Taking a distributed storage system as an example, adjustment strategies corresponding to different storage media can be pre-configured.

[0071] In some embodiments, the execution time threshold can be a pre-configured threshold related to the resource type. For example, for a certain type of RAID group, the configured execution time threshold may include gradient settings such as 100ms, 150ms, 200ms, etc.

[0072] In some embodiments, the parallel processing parameters for target type tasks may include the amount of parallel processing.

[0073] Therefore, the adjustment strategy can include the correlation between the execution duration threshold of the gradient setting and the parallel processing parameters of the target type task. For example, adjustment strategy P1 could be that when the actual execution duration is greater than 100ms, the corresponding parallel processing volume is 15. Adjustment strategy P2 could be that when the actual execution duration is greater than 150ms, the corresponding parallel processing volume is 10.

[0074] It is understandable that the longer the actual execution time of the target type task is, the less resources the current node can allocate to the target type task. Therefore, less parallel processing is needed to reduce the impact of the execution of the target type task on the execution of business tasks.

[0075] For example, when the first duration is 120ms, the target adjustment strategy can be determined to be adjustment strategy P1. Therefore, the parallel processing volume of the target type indicated by the initial resource allocation strategy can be adjusted to 15.

[0076] To verify the technical effects of the embodiments of this application, a test was conducted on the data destruction task in the distributed storage system according to the task processing method provided in the embodiments of this application. The test results show that the data destruction rate of the SSD RAID group is improved by 40-50%, the read and write stability of the HDD RAID group is improved by 35%, and the throughput of the distributed storage system is increased by 22-23%.

[0077] Therefore, by combining the characteristics of different resource types and dynamically adjusting the parallel processing volume of target type tasks, the task processing speed, system stability, and system throughput are further improved.

[0078] In addition to pre-configuring the execution duration threshold, the execution duration threshold can be represented by a predetermined execution duration threshold and multiple scaling factors to further improve the flexibility of the adjustment strategy.

[0079] Therefore, in some embodiments, determining multiple adjustment strategies associated with a resource type may include the following operations: determining a predetermined execution duration threshold associated with the resource type; and generating multiple adjustment strategies based on the predetermined execution duration threshold, multiple scaling factors, and parallel processing parameters of multiple target type tasks.

[0080] In some embodiments, the predetermined execution duration threshold can be determined based on the resource type. Multiple scaling factors can be set according to actual needs, such as 10%, 20%, 30%, etc.

[0081] In some embodiments, the parallel processing parameter can represent the amount of parallel processing, and multiple adjustment strategies are generated according to the rule that the larger the execution time threshold, the smaller the amount of parallel processing.

[0082] For example: Adjust strategy P3: 10% of the predetermined execution time threshold, corresponding to a parallel processing volume of 15; Adjust strategy P4: 20% of the predetermined execution time threshold, corresponding to a parallel processing volume of 10.

[0083] Adjustment strategies are generated based on predetermined duration thresholds corresponding to resource types and different scaling factors, along with the parallel processing parameters of the target type tasks. This allows for flexible adaptation to dynamic updates of resources on the target node. For example, for any target node in a distributed storage system, when a RAID group is added, the adjustment strategy corresponding to the storage medium of the new RAID group can be updated. This enables the distributed system to dynamically update the adjustment strategy in response to resource updates, thereby dynamically controlling the parallel processing volume of the target type tasks and further improving system stability.

[0084] In some embodiments, the plurality of adjustment strategies includes a first adjustment strategy and a second adjustment strategy. According to embodiments of this application, determining a target adjustment strategy associated with a first duration from the plurality of adjustment strategies may include the following operations: in response to determining that the first duration is greater than a first task execution duration threshold indicated by the first adjustment strategy and less than a second task execution duration threshold indicated by the second adjustment strategy, determining the first adjustment strategy as the target adjustment strategy; the first task execution duration threshold is less than the second task execution duration threshold; and in response to determining that the first duration is greater than or equal to the second task execution duration threshold, determining the second adjustment strategy as the target adjustment strategy.

[0085] For example, adjustment strategy P1 could be such that when the actual duration is greater than 100ms, the corresponding parallel processing volume is 15. Adjustment strategy P2 could be such that when the actual duration is greater than 150ms, the corresponding parallel processing volume is 10.

[0086] When the first duration is 135ms, the target adjustment strategy can be determined as adjustment strategy P1. When the first duration is 160ms, the target adjustment strategy can be determined as adjustment strategy P2.

[0087] By setting the parallel processing volume corresponding to different execution duration thresholds in the gradient settings, the parallel processing volume of the target node for the target type of task can be dynamically adjusted according to the real-time execution status of the target type of task, thereby further improving resource utilization.

[0088] In some embodiments, the parallel processing parameters include: the number of parallel processes. Adjusting the initial resource allocation strategy according to the target adjustment strategy may include the following operation: adjusting the initial number of parallel processes in the initial resource allocation strategy to the target number of parallel processes in the target adjustment strategy.

[0089] Figure 4A A schematic diagram illustrating the dynamic adjustment of resource allocation strategies according to an embodiment of this application is shown.

[0090] like Figure 4A As shown, in this embodiment 400A, the initial concurrent processing count 401 can be 5, the first duration 402 can be 35ms, and the adjustment strategy A 410A associated with the resource type is invoked. The adjustment strategy A 410A is defined as follows: T > 40ms, concurrent processing count = 5; 30 < T ≤ 40ms, concurrent processing count = 8; 0 < T ≤ 30ms, concurrent processing count = 11. T represents the actual time consumed in executing the target task.

[0091] Since the first duration 402 is 35ms, which falls within the range of 30 < T ≤ 40ms, the concurrent processing count in the target adjustment strategy can be determined to be 8. Therefore, the initial concurrent processing count indicated in the initial resource allocation strategy can be adjusted from 5 to 8, resulting in the adjusted concurrent processing count 403.

[0092] In the adjustment strategy, the number of parallel processing operations is used as the parallel processing parameter. No calculations are required, and the parallel processing parameter to be updated can be directly read from the adjustment strategy, which further improves the efficiency of dynamically adjusting the amount of parallel processing.

[0093] In a distributed system, the maximum parallel processing capacity of a target type task on any target node is pre-configured. Therefore, during dynamic adjustments, adjustments can be made based on the pre-configured maximum parallel processing capacity.

[0094] However, in practical applications, the maximum parallel processing capacity of any target node is determined based on the number of nodes in the distributed system. For example, if the maximum parallel processing capacity of a pre-configured target type task in the distributed system is 1000, and there are 4 target nodes in the distributed system, then the maximum parallel processing capacity allocated to each target node is 250. When a node is updated in the distributed system, such as adding or deleting a node, the maximum parallel processing capacity allocated to each target node will also be updated accordingly.

[0095] Therefore, the parallel processing parameters, including the ratio of the number of parallel processes of the target node to the threshold of the number of parallel processes of the target node, can be used to improve the adaptability of the adjustment strategy to node update scenarios in distributed systems.

[0096] In some embodiments, adjusting the initial resource allocation strategy according to the target adjustment strategy may include the following operations: obtaining the target parallel processing number of the target node based on the target node's parallel processing number threshold and the target ratio in the target adjustment strategy; and adjusting the initial parallel processing number of the target node to the target parallel processing number of the target node.

[0097] In some embodiments, the parallel processing quantity threshold can characterize the maximum amount of parallel processing. For example, it can be 10.

[0098] For example, adjustment strategy P5 could be such that when the actual duration is greater than 100ms, the corresponding parallel processing volume is 90% of the maximum parallel processing volume. Adjustment strategy P2 could be such that when the actual duration is greater than 150ms, the corresponding parallel processing volume is 60% of the maximum parallel processing volume.

[0099] When the first duration is 120ms and the matched target adjustment strategy is adjustment strategy P5, the corresponding target ratio can be 90%, and the number of target parallel processing can be 10×90%=9.

[0100] The initial number of parallel processes on the target node can be adjusted to 9.

[0101] Figure 4BA schematic diagram illustrating the dynamic adjustment of resource allocation strategies according to another embodiment of this application is shown.

[0102] like Figure 4B As shown, in this embodiment 400B, the initial concurrent processing count 401 can be 5, the first duration 402 can be 35ms, and the adjustment strategy B 410B associated with the resource type is invoked. The adjustment strategy B 410B is defined as follows: T > 40ms, concurrent processing count = 30% × maximum concurrent processing count; 30 < T ≤ 40ms, concurrent processing count = 50% × maximum concurrent processing count; 0 < T ≤ 30ms, concurrent processing count = maximum concurrent processing count. T represents the actual time consumed in executing the target task. For example, the maximum concurrent processing count can be 16.

[0103] Since the first duration of 35ms falls within the range of 30 < T ≤ 40ms, the concurrent processing count can be determined to be 50% × 16 = 8. Therefore, the adjusted concurrent processing count is 8.

[0104] Figure 4C A schematic diagram illustrating the dynamic adjustment of resource allocation strategies according to yet another embodiment of this application is shown.

[0105] like Figure 4C As shown, in this embodiment 400C, the initial concurrent processing count 401 can be 5, the first duration 402 can be 35ms, and the adjustment strategy C 410C associated with the resource type is invoked. The adjustment strategy C 410C is defined as follows: T > 90%Tmax, concurrent processing count = 30% × maximum concurrent processing count; 50%Tmax < T ≤ 90%Tmax, concurrent processing count = 50% × maximum concurrent processing count; 0 < T ≤ 50%Tmax, concurrent processing count = maximum concurrent processing count. T represents the actual time consumed in executing the target task. For example, the maximum concurrent processing count can be 16. Tmax represents the predetermined execution time threshold corresponding to the resource type, for example, it can be 45ms.

[0106] Since the first duration of 35ms falls within the range of 50%Tmax < T ≤ 90%Tmax, the concurrent processing count can be determined to be 50% × 16 = 8. Therefore, the adjusted concurrent processing count is 8.

[0107] The task processing method described above is executed by the first processor deployed on any target node in the distributed system. Since the initial resource allocation strategy on any target node is calculated by the second processor in the distributed system, which is used to schedule multiple target nodes, the adjustment strategy is also pre-configured. When any target node in the distributed system dynamically adjusts the parallel processing volume of a target type of task, it must adjust the parallel processing volume according to this pre-configured adjustment strategy. Therefore, to further improve the flexibility of the adjustment strategy in adapting to actual application scenarios, the pre-configured adjustment strategy can be dynamically adjusted based on the actual state of the parallel processing volume of each node in the distributed system.

[0108] Therefore, embodiments of this application provide a task processing method for a second processor deployed in a distributed system for scheduling resource configurations of multiple target nodes.

[0109] Figure 5 A task processing method applied to a second processor according to an embodiment of this application is shown.

[0110] like Figure 5 As shown, the method 500 may include operations S510~S520.

[0111] In operation S510, multiple adjustment information for multiple target nodes within a predetermined time period is obtained.

[0112] In operation S520, based on multiple adjustment information, the parallel processing parameters indicated by the adjustment strategy associated with the resource type are adjusted so that multiple target nodes can obtain target information according to the adjusted adjustment strategy.

[0113] In this embodiment of the application, the adjustment information may be obtained by multiple target nodes in a distributed system based on the task processing method described above, which is applied to the first processor of any target node.

[0114] For example, adjustment information may include, but is not limited to, adjustment frequency and adjustment direction. Adjustment frequency can refer to the number of times a target node of the same resource type adjusts its parallel processing capacity within a predetermined time period. Adjustment direction can refer to the direction in which the target node of the same resource type adjusts its parallel processing capacity within a predetermined time period, such as increasing or decreasing it.

[0115] In some embodiments, based on multiple adjustment information, the parallel processing parameters indicated by the adjustment strategy associated with the resource type are adjusted. These parallel processing parameters include, but are not limited to, the number of parallel processes and the proportion of the number of parallel processes to a threshold. This ensures that multiple target nodes each obtain target information according to the adjusted adjustment strategy, and the target information indicates the number of target type tasks that the target node processes in parallel.

[0116] For example, in a distributed system, if over 80% of the target nodes continuously reduce their parallel processing capacity within a predetermined time period, it indicates that the difference between the adjustment strategy and the parallel processing capacity corresponding to each execution duration threshold is too small, leading to frequent adjustments. Therefore, by widening the difference between the adjustment strategy and the parallel processing capacity corresponding to each execution duration threshold, the adjustment frequency can be reduced, improving the matching degree between the adjustment strategy and the application scenario, and further enhancing the stability of the system.

[0117] The following is combined with Figures 6-7 An embodiment of the above task processing method, which adjusts the parallel processing parameters indicated by an adjustment strategy associated with the resource type based on multiple adjustment information, will be described in detail.

[0118] According to an embodiment of this application, adjusting the parallel processing parameters indicated by an adjustment strategy associated with a resource type based on multiple adjustment information may include the following operations: determining the strategy adjustment frequency and strategy adjustment direction of multiple target nodes based on multiple adjustment information; determining the number of first nodes with the same strategy adjustment direction in response to determining that the strategy adjustment frequency is greater than a predetermined frequency threshold; and adjusting the multiple parallel processing parameters indicated by the adjustment strategy according to the number of first nodes and the strategy adjustment direction.

[0119] Figure 6 This illustration shows a schematic diagram of how, according to an embodiment of this application, parallel processing parameters are adjusted based on multiple adjustment information and an adjustment strategy associated with a resource type.

[0120] like Figure 6 As shown, in embodiment 600, firstly, operation S610 is performed to obtain the policy adjustment frequency and policy adjustment direction of multiple target nodes.

[0121] In real-world applications, each target node can be configured with multiple types of resources. Therefore, when calculating the frequency and direction of strategy adjustments, statistics are based on the same resource type, as the adjustment strategies for the same resource type are identical. For example, an SSD-type RAID group corresponds to adjustment strategy A.

[0122] For example, in a distributed storage system, four target nodes are configured. Within a predetermined time period, the strategy adjustment frequency of each target node based on adjustment strategy A is counted. For example, the strategy adjustment frequency of target node S1 can be 67%, the strategy adjustment frequency of target node S2 can be 70%, the strategy adjustment frequency of target node S3 can be 40%, and the strategy adjustment frequency of target node S4 can be 75%.

[0123] Then, operation S620 is executed to determine whether the policy adjustment frequency is greater than a predetermined threshold. If not, no adjustment is made.

[0124] For example, the predetermined frequency threshold can be 60%, and the nodes whose strategy adjustment frequency is greater than the predetermined frequency threshold can be identified as target node S1, target node S2 and target node S3.

[0125] Next, for target node S1, target node S2 and target node S3, perform operation S630 to determine the number of nodes whose policy adjustment direction remains the same.

[0126] For example, if the policy adjustment direction of target node S1, target node S2 and target node S3 is to continuously increase the number of parallel processing tasks of the target type, it can be determined that the number of nodes with the same policy adjustment direction is 3.

[0127] Next, operation S640 is executed to determine whether the number of nodes is greater than a first predetermined threshold. If not, no adjustment is made.

[0128] In some embodiments, adjusting multiple parallel processing parameters indicated by the adjustment strategy based on the number of first nodes and the strategy adjustment direction may include the following operation: in response to determining that the number of first nodes is greater than a first predetermined number threshold, increasing the gradient difference between the multiple parallel processing parameters indicated by the adjustment strategy.

[0129] For example, the first predetermined threshold number can be 2. The number of nodes whose policy adjustment direction remains the same is 3. Since 3 > 2, operation S650 can be executed to increase the gradient difference.

[0130] In some embodiments, gradient difference can characterize the difference in the number of concurrent processes corresponding to adjacent execution duration thresholds.

[0131] For example: the gradient difference between the number of concurrent processes corresponding to each execution duration threshold in adjustment strategy A is 3. After increasing the gradient difference, the gradient difference between the number of concurrent processes corresponding to each execution duration threshold in adjustment strategy A' is 4.

[0132] To further increase the fine-grainedness of gradient difference adjustment, in some embodiments, in response to determining that the number of first nodes is greater than a first predetermined number threshold, increasing the gradient difference between multiple parallel processing parameters indicated by the adjustment strategy may include the following operations: obtaining multiple first parameter adjustment results of multiple target nodes in a predetermined time period; determining a first adjustment coefficient based on the multiple first parameter adjustment results; and increasing the gradient difference between multiple parallel processing parameters indicated by the multiple adjustment strategies based on the first adjustment coefficient.

[0133] In some embodiments, the average rate of change of multiple first parameter adjustment results can be calculated based on a statistical algorithm, and then the average rate of change can be determined as a first adjustment coefficient, and the gradient difference can be increased based on the first adjustment coefficient.

[0134] For example, the result of adjusting the first parameter can represent a curve that continuously increases over time, and then the average curvature C of this curve can be calculated. Then, the absolute value of this average curvature can be increased by the gradient difference, for example: the increased gradient difference = the initial gradient difference × the absolute value of the average curvature C.

[0135] In addition, based on the needs of the actual scenario, other algorithms can be used to calculate the first adjustment coefficient, so that the adjusted gradient difference matches the actual parallel processing volume of each target node in the actual scenario more closely.

[0136] By statistically analyzing the actual adjustment results of each target node and dynamically updating the adjustment strategy, the adjusted gradient difference is more closely matched with the actual parallel processing volume changes of each target node in the real scenario. This reduces the frequency of resource allocation strategy adjustments for each target node and further improves the stability of the system.

[0137] In this embodiment of the application, adjusting multiple parallel processing parameters indicated by the adjustment strategy based on the strategy adjustment frequency and the strategy adjustment direction may include the following operations: in response to determining that the strategy adjustment frequency is greater than a predetermined frequency threshold, determining the number of second nodes whose strategy adjustment direction continues to fluctuate; and adjusting the multiple parallel processing parameters indicated by the adjustment strategy according to the number of second nodes and the strategy adjustment direction.

[0138] Figure 7 A schematic diagram is shown illustrating the adjustment of multiple parallel processing parameters indicated by the adjustment strategy based on the strategy adjustment frequency and the strategy adjustment direction, according to another embodiment of this application.

[0139] like Figure 7 As shown, in this embodiment 700, firstly, operation S710 is performed to obtain the policy adjustment frequency and policy adjustment direction of multiple target nodes.

[0140] In real-world applications, each target node can be configured with multiple types of resources. Therefore, when calculating the frequency and direction of strategy adjustments, statistics are based on the same resource type, as the adjustment strategies for the same resource type are identical. For example, an SSD-type RAID group corresponds to adjustment strategy A.

[0141] For example, in a distributed storage system, four target nodes are configured. Within a predetermined time period, the strategy adjustment frequency of each target node based on the adjustment strategy P for the number of parallel processes is counted. For example, the strategy adjustment frequency of target node S1 can be 67%, the strategy adjustment frequency of target node S2 can be 70%, the strategy adjustment frequency of target node S3 can be 40%, and the strategy adjustment frequency of target node S4 can be 75%.

[0142] Then, operation S720 is executed to determine whether the policy adjustment frequency is greater than a predetermined threshold. If not, no adjustment is made.

[0143] For example, the predetermined frequency threshold can be 60%, and the nodes whose strategy adjustment frequency is greater than the predetermined frequency threshold can be identified as target node S1, target node S2 and target node S3.

[0144] Next, execute operation S730 to determine the number of nodes whose strategy adjustment direction continues to fluctuate.

[0145] For example, if the policy adjustment directions of target nodes S1, S2, and S3 all fluctuate continuously, it can be determined that the number of nodes is 3.

[0146] Next, operation S740 is executed to determine whether the number of nodes is greater than the second predetermined threshold. If not, no adjustment is made.

[0147] In some embodiments, adjusting multiple parallel processing parameters indicated by the adjustment strategy according to the number of second nodes and the strategy adjustment direction may include the following operation: in response to determining that the number of second nodes is greater than a second predetermined number threshold, reducing the gradient difference between the multiple parallel processing parameters indicated by the multiple adjustment strategies.

[0148] For example: the second predetermined number threshold can be 2, the number of nodes whose policy adjustment direction fluctuates continuously is 3, 3>2, therefore, operation S750 is executed to reduce gradient difference.

[0149] In some embodiments, gradient difference can characterize the difference in the number of concurrent processes corresponding to adjacent execution duration thresholds.

[0150] For example: The gradient difference between the number of concurrent processes corresponding to each execution duration threshold in adjustment strategy A is 3. After reducing the gradient difference, the gradient difference between the number of concurrent processes corresponding to each execution duration threshold in adjustment strategy A is 2.

[0151] In some embodiments, in response to determining that the number of second nodes is greater than a second predetermined number threshold, reducing the difference between the parallel processing parameters indicated by the multiple adjustment strategies may include the following operations: obtaining multiple second parameter adjustment results of multiple target nodes in a predetermined time period; determining a second adjustment coefficient based on the multiple second parameter adjustment results; and reducing the gradient difference between the parallel processing parameters indicated by the multiple adjustment strategies based on the second adjustment coefficient.

[0152] In some embodiments, the results of adjusting multiple second parameters can be arranged in chronological order to generate a temporal sequence of the number of parallel processes. Then, a temporal neural network, such as LSTM (Long Short-Term Memory), can be used to process the temporal sequence of the number of parallel processes to generate a rate of change in the number of parallel processes. This rate of change is then used as a second adjustment coefficient to reduce gradient differences.

[0153] By statistically analyzing the actual adjustment results of each target node and dynamically updating the adjustment strategy, the adjusted gradient difference is more closely matched with the actual parallel processing volume changes of each target node in the real scenario. This reduces the frequency of resource allocation strategy adjustments for each target node and further improves the stability of the system.

[0154] In this embodiment of the application, the above method may further include the following operations: obtaining the resource allocation status of multiple target nodes in a historical time period and the historical execution duration of target type tasks; generating the resource requirements of each of the multiple target nodes in the first time period based on the historical resource allocation status and historical execution duration; and generating the initial resource allocation strategy of each of the multiple target nodes based on their respective resource requirements.

[0155] In some embodiments, resource allocation status can characterize the historical number of parallel processes for a target type task.

[0156] In some embodiments, generating the resource requirements of a target node in the first time period based on historical resource allocation status and historical execution duration may include the following operations: extracting historical resource allocation status features and historical execution duration features; and inputting the historical resource allocation status features and historical execution duration features into a trained temporal neural network to generate resource requirements.

[0157] In some embodiments, resource requirements can characterize the number of parallel processing tasks required by each target node for a target type in a future time period.

[0158] In some embodiments, the temporal neural network can be an LSTM network, which takes the number of concurrent processing samples and the sample execution duration of the target type task in time period t as input to the LSTM network, and outputs a predicted value or predicted sequence of the number of concurrent processing samples of the target type task in time period t+1. t can be an integer greater than 1. Then, any loss function suitable for deep learning algorithms, such as the cross-entropy loss function, can be used to calculate a loss value based on the predicted value or predicted sequence and the sample labels. The sample labels can be the true value or true sequence of the number of concurrent processing samples of the target type task in time period t+1. Next, the model parameters of the LSTM network are adjusted based on the loss value until the loss value converges, resulting in a trained temporal neural network.

[0159] The trained temporal neural network is used to predict the changes in the concurrent processing volume of each target node, so that the initial resource allocation strategy matches the actual load status of each target node.

[0160] In some embodiments, generating an initial resource allocation strategy based on resource requirements may include the following operations: determining the resource allocation ratio of multiple target nodes based on their resource requirements; and allocating predetermined resources according to the resource allocation ratio to generate the initial resource allocation strategy.

[0161] For example, in a distributed system, the threshold for the number of parallel processes for a target type task is 100. The resource allocation ratio for each target node, calculated based on its actual resource requirements, is 1:2:4:3. Therefore, the initial number of parallel processes allocated to each target node for the target type task in the initial resource allocation strategy is 10, 20, 40, and 30, respectively.

[0162] Resources are allocated based on the actual resource requirements of each target node, further reducing the probability of local overload.

[0163] In real-world applications, when each target node executes its assigned target type task according to its own concurrent processing capacity, some nodes may fail, causing unfinished tasks on the failed nodes to be unable to continue execution.

[0164] In view of this, the task processing method provided in the embodiments of this application may further include the following operations: in response to determining that any one of the multiple target nodes is running abnormally, obtaining the task execution status of the abnormal node; and scheduling the unfinished task of the abnormal node to an idle node among the multiple target nodes for continued execution.

[0165] For example, in a distributed system, target nodes S1, S2, and S3 are configured. The running status of each node can be monitored using heartbeat detection or status polling. When a failure is detected in target node S2, any unfinished tasks on target node S2 can be scheduled to target node S1 or S3. During task scheduling, unfinished tasks are prioritized for scheduling to idle nodes, thus ensuring the continuity of task execution throughout the distributed system.

[0166] Figure 8 A schematic diagram of a task processing device applied to a distributed system according to an embodiment of this application is shown.

[0167] like Figure 8 As shown, the task processing device 800 may include a first processor and a second processor.

[0168] The first processor can be configured in the local scheduling server (LocalScheduler, LS) of each target node in the distributed system. There can be n target nodes in the distributed system. Therefore, the corresponding local schedulers include: LS1 821, LS2 822, ..., LSn 82n.

[0169] A first processor is configured to receive an initial resource allocation strategy from a second processor; and acquire a first duration consumed by the target node in executing multiple tasks of the target type during the current time period and the resource types configured on the target node; wherein the resources configured on the target node for executing the target type tasks are determined according to the initial resource allocation strategy; the initial resource allocation strategy indicates the number of target type tasks that the target node processes in parallel; adjust the initial resource allocation strategy based on the first duration and resource types; and process multiple second tasks of the target type in parallel in subsequent time periods according to the target number information indicated by the adjusted resource allocation strategy.

[0170] The second processor can be configured in the GRM810 (Global Resource Manager) of the distributed system.

[0171] The second processor is used to send an initial resource allocation strategy to the first processor; and to obtain multiple adjustment information for adjusting the initial resource allocation strategy for multiple target nodes within a predetermined time period; and to adjust the parallel processing parameters indicated by the adjustment strategy based on the multiple adjustment information.

[0172] In some embodiments, firstly, the second processor sends an initial resource allocation policy to the first processor. Then, the first processor receives the initial resource allocation policy from the second processor and obtains the first duration consumed by the target node in executing multiple tasks of the target type during the current time period and the resource types configured on the target node.

[0173] Next, the first processor adjusts the initial resource allocation strategy based on the first duration and resource type; and processes multiple second tasks of the target type in parallel in subsequent time periods according to the target quantity information indicated by the adjusted resource allocation strategy.

[0174] Next, the system acquires adjustment information for multiple target nodes within a predetermined time period, used to adjust the initial resource allocation strategy; and based on this adjustment information, it adjusts the parallel processing parameters indicated by the adjustment strategy. This allows the second processors distributed across each target node to dynamically adjust the number of parallel processing tasks of the target type according to the updated adjustment strategy.

[0175] In this embodiment, the first processor in the local scheduling server configured at each target node can adjust the number of parallel processing tasks of the target type based on the adjustment strategy. The second processor configured in the global resource scheduling server can dynamically update the adjustment strategy according to the adjustment results at each target node, thereby realizing dynamic optimization of task scheduling and improving the overall performance stability of the distributed system.

[0176] In some embodiments, the first processor is further configured to determine a plurality of adjustment strategies associated with a resource type; the adjustment strategy indicates the relationship between an execution duration threshold and parallel processing parameters of a target type task; determine a target adjustment strategy associated with a first duration from the plurality of adjustment strategies; and adjust the initial resource allocation strategy according to the target adjustment strategy.

[0177] By combining the characteristics of different resource types, the parallel processing volume of target type tasks is dynamically adjusted, further improving task processing speed, system stability, and system throughput.

[0178] In some embodiments, the second processor is further configured to determine the policy adjustment frequency and policy adjustment direction of multiple target nodes based on multiple adjustment information; in response to determining that the policy adjustment frequency is greater than a predetermined frequency threshold, determine the number of first nodes whose policy adjustment direction remains the same; and adjust the parallel processing parameters indicated by the adjustment policy according to the number of first nodes and the policy adjustment direction.

[0179] In some embodiments, the second processor is further configured to, in response to determining that the policy adjustment frequency is greater than a predetermined frequency threshold, determine the number of second nodes whose policy adjustment direction continues to fluctuate; and adjust a plurality of parallel processing parameters indicated by the adjustment policy according to the number of second nodes and the policy adjustment direction.

[0180] By statistically analyzing the actual adjustment results of each target node and dynamically updating the adjustment strategy, the adjusted gradient difference is more closely matched with the actual parallel processing volume changes of each target node in the real scenario. This reduces the frequency of resource allocation strategy adjustments for each target node and further improves the system.

[0181] In some embodiments, to further enhance system stability and address the possibility of partial node failures in a distributed system, the task processing device of this application embodiment may further include: ARE830 (Abnormal Recovery Engine), used to detect the running status of each target node and feed it back to GRM810. This ensures that when any target node fails, GRM810 can schedule the unfinished tasks of the failed node to other nodes in the distributed system for continued execution.

[0182] It should be noted that the embodiments of this application are also applicable to virtual machine resource allocation scenarios in cloud computing environments. In a cloud computing environment, each target node can be a virtual machine within the cloud computing environment. The first processor can adjust the number of parallel processing tasks for non-business tasks in the virtual machine based on the real-time resource usage status feedback of the virtual machine. The second processor can send an initial resource allocation strategy to each virtual machine. The embodiments of this application can also be applied to other distributed systems, which will not be elaborated here.

[0183] Figure 9 A block diagram of an electronic device suitable for implementing a task processing method according to an embodiment of this application is shown.

[0184] like Figure 9 As shown, an electronic device 900 according to an embodiment of this application includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0185] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0186] According to embodiments of this application, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0187] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0188] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0189] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the task processing method provided in the embodiments of this application.

[0190] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0191] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0192] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0193] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0194] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0195] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0196] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A task processing method characterized by, A first processor applied to any target node deployed in a distributed system; the method comprises: obtaining a first duration consumed by a target node in a current period for executing a plurality of first tasks of a target type and a resource type configured on the target node; wherein the resources configured on the target node for executing the target type tasks are determined according to an initial resource allocation strategy; the initial resource allocation strategy indicates initial quantity information of the target node for processing the target type tasks in parallel; the target type task is a non-business type task; and determining a predetermined execution duration threshold associated with the resource type; generating a plurality of adjustment strategies according to the predetermined execution duration threshold, a plurality of proportion coefficients and a plurality of parallel processing parameters of the target type tasks; wherein the adjustment strategy indicates the association between the execution duration threshold and the parallel processing parameter of the target type task; determining a target adjustment strategy associated with the first duration from the plurality of adjustment strategies; adjusting the initial resource allocation strategy according to the target adjustment strategy; so that the second processor deployed in the distributed system for scheduling the resource configuration of the plurality of target nodes adjusts the parallel processing parameter indicated by the adjustment strategy associated with the resource type based on a plurality of adjustment information of the plurality of target nodes for adjusting the initial resource allocation strategy within a predetermined period, so that each of the plurality of target nodes obtains target information according to the adjusted adjustment strategy, and the target information indicates the number of the target type tasks processed in parallel by the target node; the target node processes a plurality of second tasks of the target type in parallel according to the target quantity information indicated by the adjusted resource allocation strategy in a subsequent period.

2. The method of claim 1, wherein, The plurality of adjustment strategies comprises a first adjustment strategy and a second adjustment strategy; The determining a target adjustment strategy associated with the first duration from the plurality of adjustment strategies comprises: in response to determining that the first duration is greater than a first task execution duration threshold indicated by the first adjustment strategy and less than a second task execution duration threshold indicated by the second adjustment strategy, determining that the first adjustment strategy is the target adjustment strategy; the first task execution duration threshold is less than the second task execution duration threshold; and in response to determining that the first duration is greater than or equal to the second task execution duration threshold, determining that the second adjustment strategy is the target adjustment strategy.

3. The method of claim 2, wherein, The parallel processing parameter comprises a parallel processing quantity; The adjusting the initial resource allocation strategy according to the target adjustment strategy comprises: adjusting the initial parallel processing quantity in the initial resource allocation strategy to a target parallel processing quantity in the target adjustment strategy.

4. The method of claim 2, wherein, The parallel processing parameter comprises a proportion of the parallel processing quantity of the target node to a parallel processing quantity threshold of the target node; The adjusting the initial resource allocation strategy according to the target adjustment strategy comprises: for any target node, obtaining the target parallel processing quantity of the target node according to the parallel processing quantity threshold of the target node and the target proportion in the target adjustment strategy; adjusting an initial parallel processing quantity of the target node to a target parallel processing quantity of the target node. 5.A task processing method applied to a second processor deployed in a distributed system for scheduling resource configurations of a plurality of target nodes; the method comprising: obtaining a plurality of adjustment information of the plurality of target nodes within a predetermined period; wherein the plurality of adjustment information is obtained based on the method of any one of claims 1-4; and based on the plurality of adjustment information, adjusting parallel processing parameters indicated by an adjustment strategy associated with a resource type, so that each of the plurality of target nodes obtains target information according to the adjusted adjustment strategy, the target information indicating a number of target nodes processing a target type task in parallel.

6. The method of claim 5, wherein, The method further comprises: based on the plurality of adjustment information, determining a strategy adjustment frequency and a strategy adjustment direction of the plurality of target nodes; in response to determining that the strategy adjustment frequency is greater than a predetermined frequency threshold, determining that the strategy adjustment direction is consistent for a first number of nodes; and adjusting the plurality of parallel processing parameters indicated by the adjustment strategy according to the first number of nodes and the strategy adjustment direction.

7. The method of claim 6, wherein, The method further comprises: in response to determining that the first number of nodes is greater than a first predetermined number threshold, increasing a gradient difference between the plurality of parallel processing parameters indicated by the adjustment strategy.

8. The method of claim 7, wherein, The method further comprises: obtaining a plurality of first parameter adjustment results of the plurality of target nodes within a predetermined period; based on the plurality of first parameter adjustment results, determining a first adjustment coefficient; and based on the first adjustment coefficient, increasing the gradient difference between the plurality of parallel processing parameters indicated by the plurality of adjustment strategies.

9. The method of claim 6, wherein, The method further comprises: in response to determining that the strategy adjustment frequency is greater than a predetermined frequency threshold, determining that the strategy adjustment direction is consistent for a second number of nodes; and adjusting the plurality of parallel processing parameters indicated by the adjustment strategy according to the second number of nodes and the strategy adjustment direction.

10. The method of claim 9, wherein, The method further comprises: in response to determining that the second number of nodes is greater than a second predetermined number threshold, reducing the gradient difference between the plurality of parallel processing parameters indicated by the plurality of adjustment strategies.

11. The method of claim 9, wherein, The method further comprises: obtaining a plurality of second parameter adjustment results of the plurality of target nodes within a predetermined period; based on the plurality of second parameter adjustment results, determining a second adjustment coefficient; and based on the second adjustment coefficient, reducing the gradient difference between the plurality of parallel processing parameters indicated by the plurality of adjustment strategies. The second adjustment coefficient is used to reduce the gradient difference between the parallel processing parameters indicated by the plurality of adjustment strategies.

12. The method of claim 5, wherein, The method further comprises: obtaining historical resource allocation states of the plurality of target nodes in a historical period and historical execution durations of the target type tasks; based on the historical resource allocation states and the historical execution durations, generating resource demands of the plurality of target nodes respectively in a first period; and based on the resource demands of the plurality of target nodes respectively, generating initial resource allocation strategies of the plurality of target nodes respectively.

13. The method of claim 12, wherein, The method further comprises: extracting historical resource allocation state features and historical execution duration features; and inputting the historical resource allocation state features and the historical execution duration features into a trained time sequence neural network to generate the resource demands.

14. The method of claim 12, wherein, The method further comprises: based on the resource demands of the plurality of target nodes, determining resource allocation proportions of the plurality of target nodes; and allocating predetermined resources according to the resource allocation proportions to generate the initial resource allocation strategies.

15. The method of claim 14, wherein, The method further comprises: in response to determining that any node of the plurality of target nodes is running abnormally, obtaining a task execution state of the abnormal node; and scheduling tasks that have not been executed by the abnormal node to idle nodes of the plurality of target nodes for continuous execution.

16. A task processing device applied to a distributed system, characterized by comprising: The device comprises: a first processor configured to receive an initial resource allocation strategy from a second processor, obtain a first duration consumed by a target node in a current period for executing a plurality of tasks of a target type, and obtain a resource type configured on the target node; wherein the resource configured on the target node for executing the target type tasks is determined according to the initial resource allocation strategy; the initial resource allocation strategy indicates quantity information of the target type tasks processed in parallel by the target node; the initial resource allocation strategy is adjusted based on the first duration and the resource type; and the target node processes a plurality of second tasks of the target type in parallel in a subsequent period according to quantity information of the target type tasks indicated by the adjusted resource allocation strategy; the target type tasks are non-business type tasks; a second processor configured to send the initial resource allocation strategy to the first processor, obtain a plurality of adjustment information of the plurality of target nodes in a predetermined period, and adjust parallel processing parameters indicated by the adjustment strategies based on the plurality of adjustment information; wherein the adjustment information is obtained based on the method of any one of claims 1-4.

17. The apparatus of claim 16, wherein, The first processor is further configured to: determine a plurality of adjustment strategies associated with the resource type; the adjustment strategies indicate an association relationship between an execution duration threshold and parallel processing parameters of the target type tasks; determine a target adjustment strategy associated with the first duration from the plurality of adjustment strategies; and adjust the initial resource allocation strategy according to the target adjustment strategy.

18. The apparatus of claim 16, wherein, The second processor is further configured to: based on the plurality of adjustment information, determine a strategy adjustment frequency and a strategy adjustment direction of the plurality of target nodes; and in response to determining that the policy adjustment frequency is greater than a predetermined frequency threshold, determining a policy adjustment direction for a first number of consecutive nodes; and adjusting a parallel processing parameter indicated by the adjustment policy in accordance with the first number of consecutive nodes and the policy adjustment direction.

19. The apparatus of claim 16, wherein, The second processor is further configured to: in response to determining that the policy adjustment frequency is greater than a predetermined frequency threshold, determining a policy adjustment direction for a second number of fluctuating nodes; and adjusting a plurality of parallel processing parameters indicated by the adjustment policy in accordance with the second number of fluctuating nodes and the policy adjustment direction.

20. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, The computer program or instructions, when executed by a processor, implement the steps of the method according to any one of claims 1-15.

Images