Methods, related devices and computer program products for processing tasks

Abstract

This application provides a method, related apparatus, and computer program product for processing tasks. In response to receiving a task scheduling request from a scheduling node, the application reads the storage load of the target storage used locally, wherein the storage load is collected by a resource collection tool running locally. In response to the storage load being less than or equal to a first load threshold, the application receives the target task corresponding to the task scheduling request from the scheduling node. The target task is then processed locally using the target storage. This enables the computing node to perceive the disk storage load in real time and dynamically adjust its task receiving strategy accordingly, thereby avoiding task stagnation caused by disk congestion and improving system resource utilization and overall throughput.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device, computer-readable medium, and computer program product for processing tasks. Background Technology

[0002] With the rapid development of information technology, the amount of data generated by various applications has exploded, and the demand for computing and storage resources has far exceeded the processing limits of single-machine devices. To cope with the challenges of processing massive amounts of data and high-concurrency tasks, distributed system architecture has emerged and become the mainstream technical solution.

[0003] Distributed systems achieve efficient processing of large-scale computing and massive amounts of data by distributing computing tasks and data storage across multiple interconnected computing nodes. Consequently, in fields such as cloud computing and big data, distributed systems have become the core infrastructure supporting modern internet services and enterprise-level applications.

[0004] Against this backdrop, ensuring the stability of distributed systems and improving their operational efficiency and overall throughput are issues that deserve attention and are urgently needed. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for processing tasks, which can be applied to computing nodes to enable computing nodes to perceive disk storage load in real time and dynamically adjust task receiving strategies accordingly, thereby avoiding task stagnation caused by disk blockage and improving system resource utilization and overall throughput.

[0006] One aspect of this application provides a method for processing tasks, applied to a computing node, comprising: in response to receiving a task scheduling request sent by a scheduling node, reading the storage load of a target storage used locally, wherein the storage load is collected by a resource collection tool running locally; in response to the storage load being less than or equal to a first load threshold, receiving a target task corresponding to the task scheduling request from the scheduling node; and processing the target task locally using the target storage.

[0007] In another aspect, this application provides an apparatus for processing tasks, applied to a computing node, comprising: a load reading module configured to read the storage load of a target storage used locally in response to receiving a task scheduling request sent by a scheduling node, wherein the storage load is collected by a resource collection tool running locally; a task receiving module configured to receive a target task corresponding to the task scheduling request from the scheduling node in response to the storage load being less than or equal to a first load threshold; and a task processing module configured to process the target task locally using the target storage.

[0008] In another aspect of this application, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the processing task provided above.

[0009] Another aspect of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to perform the processing task provided above.

[0010] In another aspect of this application, there is a computer program product including a computer program having computer program instructions stored thereon, which, when executed by a processor, can perform the processing task as provided above.

[0011] In the solution provided in this application embodiment, firstly, in response to receiving a task scheduling request sent by the scheduling node, the storage load of the target storage used locally is read, wherein the storage load is collected by a resource collection tool running locally; then, in response to the storage load being less than or equal to a first load threshold, the target task corresponding to the task scheduling request is received from the scheduling node; finally, the target task is processed locally using the target storage. This enables the computing node to perceive the disk storage load in real time and dynamically adjust the task receiving strategy accordingly, thereby avoiding task stagnation caused by disk congestion and improving system resource utilization and overall throughput. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0014] Figure 1 A flowchart illustrating a task processing procedure provided in one embodiment of this application;

[0015] Figure 2 A flowchart illustrating another process for processing a task provided in one embodiment of this application;

[0016] Figure 3 An architecture diagram illustrating the process for implementing a task in a specific application scenario, as provided in an embodiment of this application;

[0017] Figure 4 A schematic diagram of the structure of a device for processing tasks provided in an embodiment of this application;

[0018] Figure 5 This is a schematic diagram of the structure of an electronic device suitable for implementing the solutions in the embodiments of this application.

[0019] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] In a typical configuration of this application, the terminal and the service network devices each include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0022] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0023] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer program instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0024] As discussed above, ensuring the stability of distributed systems and improving their operating efficiency and overall throughput are issues that deserve attention and are urgently needed.

[0025] In some solutions, to ensure the stable and smooth operation of a distributed system, the nodes in the distributed system can be specifically divided into two types: compute nodes and scheduling nodes. Then, the scheduling node is chosen to receive the tasks to be processed. After receiving the tasks, based on the computing performance and resources of the compute nodes (e.g., the performance of the central processing unit and graphics processing unit, and the number of available threads), the task to be processed is assigned to a compute node, and the compute node is used to actually process the task.

[0026] However, in this approach, the decision-making and scheduling mainly rely on the performance and resource usage of computing nodes, such as the central processing unit and graphics processing unit. These are used as the basis for task allocation and scheduling without fully considering the storage load of computing nodes. This can lead to input / output (I / O) blockage when the distributed system is processing tasks that require frequent read and write operations, such as audio and video transcoding. This can result in the task execution process being "stuck" or stalled, wasting computing resources and reducing the overall throughput efficiency of the system.

[0027] To address this issue, this application provides a method for processing tasks. First, in response to receiving a task scheduling request from a scheduling node, the method reads the storage load of the target storage used locally, wherein the storage load is collected by a resource collection tool running locally. Then, in response to the storage load being less than or equal to a first load threshold, the method receives the target task corresponding to the task scheduling request from the scheduling node. Finally, the method processes the target task locally using the target storage. This enables the computing node to perceive the disk storage load in real time and dynamically adjust its task receiving strategy accordingly, thereby avoiding task stagnation caused by disk congestion and improving system resource utilization and overall throughput.

[0028] In real-world scenarios, the entity executing this method can be a "computing node" in a distributed system. Typically, this computing node can be represented as a network device within the distributed system. Network devices include, but are not limited to, implementations such as network hosts, single network servers, sets of multiple network servers, or cloud-based computer clusters.

[0029] In some scenarios, the computing node can also be a user device, or a device formed by integrating user devices and network devices through a network. User devices include, but are not limited to, various terminal devices such as computers, mobile phones, tablets, smartwatches, and wristbands.

[0030] In some scenarios, the executing entity can also be an application running on the aforementioned network devices or user devices. When the executing entity is software in the form of an application, it can be installed on the electronic devices listed above. It can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitations are made here.

[0031] Figure 1 The present application illustrates a process 100 for processing a task, which includes at least the following processing steps:

[0032] (Step) S101, in response to receiving a task scheduling request sent by the scheduling node, read the storage load of the target storage used locally;

[0033] In embodiments of this application, for example, the execution entity of the "computing node" in the distributed system described above can receive a task scheduling request sent by the "scheduling node" in the distributed system.

[0034] For example, after receiving a target task to be processed provided by a user device, the scheduling node can decide on a computing node (i.e., the "executing entity" mentioned above) based on a predetermined scheduling order or based on the performance of the central processing unit, graphics processing unit, etc.

[0035] Then, the scheduling node sends a task scheduling request to the execution entity, requesting that the "target task to be processed" be scheduled to the execution entity and that the execution entity be used to process the target task.

[0036] It should be understood that in some scenarios, if there is no computing node acting as a scheduling node in the distributed system, for example, if the user's use of the distributed system is achieved by the user device directly sending requests to the computing node, then in such cases, since the user device actually selects and determines the computing node, the user device that provides and sends the task can also be understood as the "scheduling node".

[0037] The target task to be processed can take different forms depending on the scenario. For example, it can be an image processing task, an audio processing task, or an audio-video transcoding task as discussed above. For instance, the target task could be a video transcoding task that converts an already compressed video stream into another video stream to adapt to different network bandwidths, different terminal processing capabilities, and different user needs.

[0038] Accordingly, in this step, if the executing entity receives the task scheduling request, it can first select to read the storage load of the target storage used locally for executing and processing the target task.

[0039] The target storage can be pre-connected to the execution entity (or deployed locally on the execution entity) and can be used by the execution entity to store relevant data of the target task, intermediate data generated during the processing of the target task, and the final processing result of the target task. Accordingly, the storage load of the target storage can be, for example, the input / output (I / O) utilization rate of the target storage.

[0040] The storage load of the target storage can be collected by a resource collection tool deployed and running locally on the execution entity. The resource collection tool can be pre-installed and loaded into the compute node (i.e., the "local" location of the execution entity) so that it can continuously count and collect the current storage load of the target storage currently used by the compute node, so as to dynamically and in real time maintain and record the storage load.

[0041] Typically, this resource collection tool can be set and configured to collect and acquire storage load in a "periodic" manner, so that the resource collection tool can stably maintain the storage load.

[0042] In some embodiments, the resource collection tool can be configured to acquire and statistically analyze storage load each time the execution entity starts executing a new pending task and / or completes processing a pending task. This configuration allows the storage load maintained by the resource collection tool to be correlated and matched with the specific task execution status of the computing nodes, enabling the resource collection tool to achieve timely and highly correlated accurate collection of storage load at a lower cost and with fewer computing resources.

[0043] In some embodiments, since there may be multiple computing nodes in a distributed system, each computing node can be configured with a resource collection tool to record and maintain the storage load of the target storage used by that computing node, so that similar management and processing methods can be implemented for all these computing nodes. In such a case, to reduce the deployment difficulty of the distributed system and save deployment resources, at least two or more computing nodes can choose to use a "shared storage" (e.g., it can be "shared memory") to store their respective storage loads. Accordingly, in such a case, the resource storage tool in each computing node can collect the storage load of the corresponding computing node for the storage (target storage) of the computing node, and store and maintain the storage load to the "shared storage". Thus, the storage load of the target storage of each computing node can be maintained as a whole using the "shared storage", thereby simplifying the structure of the distributed system.

[0044] In some embodiments, if the resource storage tool in the execution entity uses the aforementioned shared storage to record and store storage load, then during the process of reading the storage load of the target storage used locally, the execution entity may, as an alternative, choose to read the storage load of the target storage used locally from the shared storage to complete the aforementioned storage load reading purpose.

[0045] Next, if the storage load is less than or equal to the first load threshold (for example, the first load threshold can be set in advance based on the standard that the target storage has a low storage load, sufficient storage space, and is not likely to be blocked), the execution entity can respond to this by executing S102 to receive the target task to be processed corresponding to the task scheduling request from the scheduling node.

[0046] S102, Receive the target task corresponding to the task scheduling request from the scheduling node;

[0047] In some embodiments, in order to manage and schedule tasks more effectively, it is also possible to configure corresponding task properties based on the specific circumstances of the task, so that the executing entity can make a better "decision" on whether to actually receive the target task corresponding to the task scheduling request from the scheduling node based on the task properties.

[0048] For example, tasks with high storage sensitivity and high I / O read performance requirements can be marked as anti-affinity tasks (e.g., using task nature tags). This allows the execution entity to control the execution of only one such anti-affinity task at a time, ensuring and prioritizing the fulfillment of its storage sensitivity and I / O read performance requirements. Alternatively, anti-affinity scheduling can be used to ensure that an execution entity executes only one such task at a time.

[0049] Accordingly, in such a case, during the process of receiving the target task corresponding to the task scheduling request from the scheduling node, the executing entity may choose to first read the task nature of the target task corresponding to the task scheduling request.

[0050] Then, if the target task is an anti-affinity task, the executing entity can respond by determining whether other existing tasks that are anti-affinity tasks are currently being executed locally.

[0051] If there are no other existing tasks that are anti-affinity tasks currently being executed locally, the executing entity can respond to this, that is, in response to the fact that the task is an anti-affinity task and there are no other existing tasks that are anti-affinity tasks currently being executed locally, by selecting to receive the target task corresponding to the task scheduling request from the scheduling node.

[0052] This enables distributed systems to schedule and process tasks with special storage resource requirements more efficiently and with higher quality.

[0053] Accordingly, if other tasks that are anti-affinity tasks are being executed locally, the executing entity can respond by controlling and notifying the scheduling node to select other computing nodes to handle the target task, and refusing to handle the target task locally. This achieves the aforementioned anti-affinity scheduling.

[0054] In some embodiments, if a large number of tasks with anti-affinity characteristics exist among the tasks currently to be executed and scheduled, the executing entity may choose to temporarily allow the processing of two or more such tasks locally to improve computational efficiency and alleviate the task processing pressure on the distributed system. For example, when the number of anti-affinity tasks is greater than or equal to a threshold, the task scheduling node can add an indicator to indicate to the executing entity that the number of anti-affinity tasks is "large". Accordingly, after receiving the indicator, the executing entity can further determine, based on its storage load, whether it can still support processing this new anti-affinity task while processing the current anti-affinity tasks (e.g., whether processing the new anti-affinity task will cause the storage load to exceed a first load threshold). If it can (i.e., it will not exceed the first load threshold), the executing entity can respond by choosing to accept the new anti-affinity task to alleviate the pressure on the distributed storage system.

[0055] S103 utilizes target storage to process target tasks locally.

[0056] In the embodiments of this application, if the executing entity chooses to receive the target task in S102, then in this step, the executing entity can use the target storage to execute the target task locally after receiving it. For example, the executing entity can complete the video transcoding process described above locally using the target storage.

[0057] The task processing method provided in this application firstly reads the storage load of the target storage used locally in response to receiving a task scheduling request from the scheduling node, wherein the storage load is collected by a resource collection tool running locally; then, in response to the storage load being less than or equal to a first load threshold, the target task corresponding to the task scheduling request is received from the scheduling node; finally, the target task is processed locally using the target storage. This enables the computing node to perceive the disk storage load in real time and dynamically adjust the task receiving strategy accordingly, thereby avoiding task stagnation caused by disk congestion and improving system resource utilization and overall throughput.

[0058] In some embodiments, if the current storage load of the target storage cannot support the processing of the target task, for example, in S102 above, the storage load read by the execution subject is greater than the first load threshold, the execution subject may respond by controlling the scheduling node to schedule the target task to other computing nodes, so as to avoid the computing nodes using the target storage to process the target task locally, which would cause the target storage to become blocked and ensure the working performance of the target storage.

[0059] In some embodiments, in order to improve the computing efficiency and task scheduling efficiency of the distributed system, a second load threshold with a value greater than the first load threshold can also be set.

[0060] Typically, the second load threshold can be set based on the criterion that the "storage load" will not be released in the short term. In some embodiments, the "short term" can be determined based on the processing time of the "task" after determining the difference between the second load threshold and the first load threshold. Accordingly, after the execution entity reads the storage load based on S101, it can further compare it with the second load threshold if the load is greater than the first load threshold.

[0061] If the storage load is greater than the first load threshold but less than or equal to the second load threshold, it indicates that the storage load exceeding the first load threshold can be mitigated in the short term. In this case, the execution entity can choose to control the scheduling node to schedule the target task to other computing nodes. This allows the scheduling node to "try" or "probe" whether the execution entity can accept a new task. This ensures that new tasks can be scheduled to the execution entity as early as possible, improving the overall throughput of the distributed system.

[0062] If the storage load exceeds the second load threshold, the execution entity can choose to respond by first controlling the scheduling node to schedule the target task to other computing nodes.

[0063] Then, the scheduling node is prohibited from sending new task scheduling requests locally. This is to avoid situations where the scheduling node is unable to effectively schedule new tasks due to frequent attempts and rejections between the executor and the scheduling node, thereby improving the overall task scheduling capability of the distributed system while avoiding wasting scheduling resources.

[0064] To better understand situations involving such processes, you can also refer to... Figure 2 . Figure 2 The present application illustrates another processing procedure 200 for a processing task, which includes at least the following processing steps:

[0065] S201, in response to receiving a task scheduling request sent by the scheduling node, read the storage load of the target storage used locally;

[0066] As discussed above, if the storage load is less than or equal to the first load threshold, the execution entity can respond by executing S202.

[0067] S202, Receive the target task corresponding to the task scheduling request from the scheduling node;

[0068] S203 utilizes target storage to process target tasks locally.

[0069] The above S201-S203 are due to actual differences and such Figure 1 The S101-103 shown are the same. For the same parts, please refer to the corresponding parts of the previous embodiment. They will not be repeated here.

[0070] In process 200 shown in this embodiment, after S201, if the storage load is greater than the first load threshold, the execution entity may respond by choosing to execute S204 instead of S202.

[0071] As discussed above, if the storage load is only greater than the first load threshold and less than or equal to the second load threshold, the execution entity can choose to terminate process 200 after completing execution S204.

[0072] Furthermore, as discussed above, if the storage load is still greater than the second load threshold, the execution entity can choose to respond by executing S204 and then continuing to execute S205, prohibiting the scheduling node from sending new task scheduling requests locally.

[0073] S205, prohibits the scheduling node from sending new task scheduling requests to the local machine.

[0074] It should be understood that the executing entity can actually choose to directly compare the storage load with the first load threshold and the second load threshold in, for example, S201, and choose whether to execute S202 and S203 continuously, or execute S204, or execute S204 and S205 continuously based on the comparison result.

[0075] Alternatively, the storage load can be compared with the first load threshold in a stepwise manner. Then, if the storage load is greater than the first load threshold and S204 has been executed, it can be compared with the second load threshold to determine whether to continue executing S205 or exit and end process 200.

[0076] In some embodiments, in order to more effectively maintain the usage status of the target storage and avoid prolonged failures or inability to recover due to blockage, a third load threshold numerically greater than the second load threshold can be further set so that the executing entity can use the third load threshold to determine whether the target storage is about to be "blocked" with a high probability (for example, the third load threshold may correspond to the "allowed usage limit" of the target storage with a low probability of being "blocked").

[0077] That is, if the executing entity can, after acquiring the storage load, as discussed above,

[0078] The storage load is compared simultaneously with the first load threshold, the second load threshold, and the third load threshold, or, after executing S205, the storage load is further compared with the third load threshold.

[0079] Regardless of the method described above, if the executing entity determines that the storage load exceeds the third load threshold, it can respond by immediately stopping any currently executing tasks and triggering failure callbacks for those tasks. In other words, if the executing entity judges that the current storage load on the target storage has a high probability of causing it to become "blocked," it can immediately release the storage load by triggering failure callbacks for currently executing tasks. This instantly alleviates the storage pressure on the target storage, preventing it from experiencing prolonged failures or becoming unusable due to blockage. Correspondingly, this method also improves task execution stability by promptly releasing currently executing tasks to prevent them from "freezing" due to target storage blockage.

[0080] In some embodiments, if the executing entity chooses to release the resources and storage load of the target storage it uses in such a way, then in this case, the executing entity may also choose to prohibit the scheduling node from sending new task scheduling requests locally, so as to ensure that the storage load is reliably reduced.

[0081] Accordingly, in such a case, the execution order of the executing entity's actions to stop the execution of existing tasks and to prevent the scheduling node from sending new task scheduling requests to the local machine can differ in different embodiments. That is, in different embodiments, for the two actions of "stopping the existing task being executed locally and triggering the failure callback of the existing task" and "preventing the scheduling node from sending new task scheduling requests to the local machine," the executing entity can arbitrarily choose one of them to be executed first, or execute both simultaneously. For example, in different embodiments, if it is necessary to execute S205 and S206 (for example, after S201, the storage load is compared with the first load threshold, the second load threshold, and the third load threshold, and it is determined that the storage load is greater than the third load threshold), the executing entity can choose to execute S205 first and then execute S206 below, or execute S206 below and then execute S205, or execute S205 and S206 simultaneously.

[0082] For ease of understanding, refer to process 200 and consider the scenario where S205 is executed before S206. For example, in process 200, the execution entity can choose to execute S206 after S205 if the storage load exceeds the third load threshold.

[0083] S206, Stop any existing task currently being executed locally and trigger the failure callback of the existing task.

[0084] In some embodiments, after prohibiting the scheduling node from sending new task scheduling requests locally, the execution entity may also choose to periodically detect changes in storage load. Typically, if the resource collection tool collects data "periodically," the period used here can be comparable to or longer than the resource collection tool's collection period to ensure that the execution entity can perform detection only after the resource collection tool provides new storage load information, thus avoiding wasting computing resources.

[0085] If the changed storage load is less than or equal to the first load threshold, the execution entity can choose to respond accordingly and resume allowing the scheduling node to send new task scheduling requests locally. This enables compute nodes to adaptively go offline and online based on the actual storage load, thereby ensuring the reliability of the distributed system while reducing its management costs and improving its task throughput.

[0086] To enhance understanding, this application also provides an exemplary architecture for implementing a task processing procedure, using a specific application scenario as an example. Please refer to [reference needed]. Figure 3 , Figure 3This is an architecture diagram of an architecture 300 for implementing a process of processing tasks in a specific application scenario, provided as an embodiment of this application.

[0087] As discussed above, architecture 300 can actually be an "architecture" of a distributed system, which may include at least scheduling node 301, and computing nodes 320, 330... 3N0 (where N is a positive integer).

[0088] Accordingly, each compute node may include a resource collection tool for collecting and acquiring the storage load of the storage corresponding to the compute node. For example, compute node 320 may include a resource collection tool 321 for collecting the storage load 323 of storage 322, compute node 330 may include a resource collection tool 331 for collecting the storage load 333 of storage 332, and compute node 3N0 may include a resource collection tool 3N1 for collecting the storage load 3N3 of storage 3N2.

[0089] For example, in architecture 300, each computing node can jointly use "shared storage 302" to maintain and store the "storage load" collected by its respective resource collection tools. For example, shared storage 302 can be used to maintain and record the storage load 323 of storage 322, the storage load 333 of storage 332, ... and the storage load 3N3 of storage 3N2.

[0090] This application also provides an apparatus for processing tasks, the structure of which is as follows: Figure 4 The apparatus 400 shown includes: a load reading module 410 configured to read the storage load of the target storage used locally in response to receiving a task scheduling request sent by a scheduling node, wherein the storage load is collected by a resource collection tool running locally; a task receiving module 420 configured to receive a target task corresponding to the task scheduling request from the scheduling node in response to the storage load being less than or equal to a first load threshold; and a task processing module 430 configured to process the target task locally using the target storage.

[0091] This embodiment exists as a device embodiment corresponding to the above method embodiment. The device for processing tasks provided in this embodiment enables computing nodes to perceive disk storage load in real time and dynamically adjust task receiving strategies accordingly, thereby avoiding task stagnation caused by disk blockage and improving system resource utilization and overall throughput.

[0092] In some embodiments, the apparatus 400 further includes: a first control module configured to control a scheduling node to schedule the target task to other computing nodes in response to a storage load being greater than a first load threshold and less than or equal to a second load threshold, wherein the value of the second load threshold is greater than the first load threshold.

[0093] In some embodiments, the apparatus 400 further includes: a second control module configured to control the scheduling node to schedule the target task to other computing nodes in response to the storage load being greater than a second load threshold; and a first scheduling prohibition module configured to prohibit the scheduling node from sending new task scheduling requests to the local machine.

[0094] In some embodiments, the apparatus 400 further includes: a task stop module configured to stop an existing task being executed locally and trigger a failure callback of the existing task in response to a storage load exceeding a third load threshold, wherein the value of the third load threshold is greater than a second load threshold; and a second scheduling prohibition module configured to prohibit scheduling nodes from sending new task scheduling requests to the local machine.

[0095] In some embodiments, the apparatus 400 further includes: a load change detection module configured to periodically detect changes in storage load after prohibiting the scheduling node from sending new task scheduling requests locally; and a scheduling recovery module configured to restore permission for the scheduling node to send new task scheduling requests locally in response to the changed storage load being less than or equal to a first load threshold.

[0096] In some embodiments, the task nature of the target task corresponding to the task scheduling request is read; in response to the task nature being an anti-affinity task and no other existing tasks belonging to the anti-affinity task are currently being executed locally, the target task corresponding to the task scheduling request is received from the scheduling node.

[0097] In some embodiments, after collecting the storage load, the resource storage tool stores the storage load in a shared storage maintained by at least two computing nodes, and reads the storage load of the target storage used locally, including: reading the storage load of the target storage used locally from the shared storage.

[0098] Based on the same concept, this application also provides an electronic device, a readable storage medium, and a computer program product. The method corresponding to the electronic device can be the task processing method in the foregoing embodiments, and its problem-solving principle is similar to that method. The electronic device provided in this application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods and / or technical solutions of the foregoing embodiments of this application.

[0099] Electronic devices can be user devices, or devices composed of user devices and network devices integrated through a network, or applications running on the aforementioned devices. User devices include, but are not limited to, various terminal devices such as computers, mobile phones, tablets, smartwatches, and wristbands. Network devices include, but are not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, and can be used to implement some processing functions when setting an alarm clock. Here, the cloud consists of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.

[0100] Figure 5 The diagram illustrates the structure of an electronic device suitable for implementing the methods and / or technical solutions in the embodiments of this application. The electronic device 500 includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes based on a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage portion 508 into a Random Access Memory (RAM) 503. The RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An Input / Output (I / O) interface 505 is also connected to the bus 504.

[0101] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, touchscreen, microphone, infrared sensor, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), LED display, OLED display, etc., and speakers, etc.; a storage section 508 including one or more computer-readable media such as hard disk, optical disk, magnetic disk, semiconductor memory, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet.

[0102] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 501, it performs the functions defined in the methods of this application.

[0103] Another embodiment of this application provides a computer-readable storage medium and a computer program product having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0104] Specifically, this embodiment may employ any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, a system, apparatus, or device that is, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0105] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0106] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0107] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0108] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0109] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules and units is only a logical functional division, and in actual implementation, there may be other division methods. Taking units as examples, multiple units or page components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0111] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] Furthermore, the functional modules and units in the various embodiments of this application can be integrated into one processing module or unit, or each module or unit can exist physically separately, or two or more units can be integrated into one module or unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules and units.

[0113] The integrated modules and units implemented as software functional modules and units described above can be stored in a computer-readable storage medium. These software functional modules and units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0115] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A method for processing tasks, applied to a computing node, comprising: In response to receiving a task scheduling request from a scheduling node, the storage load of the target storage used locally is read, wherein the storage load is collected by a resource collection tool running locally; In response to the storage load being less than or equal to a first load threshold, the target task corresponding to the task scheduling request is received from the scheduling node; The target task is processed locally using the target storage.

2. The method of claim 1, wherein, The method further includes: In response to the storage load being greater than a first load threshold and less than or equal to a second load threshold, the scheduling node is controlled to schedule the target task to other computing nodes, wherein the value of the second load threshold is greater than the first load threshold.

3. The method of claim 2, wherein, The method further includes: In response to the storage load exceeding the second load threshold, the scheduling node is controlled to schedule the target task to other computing nodes; The scheduling node is prohibited from sending new task scheduling requests to the local machine.

4. The method of claim 2, wherein, The method further includes: In response to the storage load exceeding a third load threshold, the existing task currently being executed locally is stopped, and a failure callback for the existing task is triggered, wherein the value of the third load threshold is greater than the second load threshold. The scheduling node is prohibited from sending new task scheduling requests to the local machine.

5. The method according to claim 3 or 4, characterized in that, The method further includes: After prohibiting the scheduling node from sending new task scheduling requests locally, the changes in the storage load are periodically detected; In response to the changed storage load being less than or equal to the first load threshold, the scheduling node is allowed to send new task scheduling requests locally again.

6. The method of claim 1, wherein, Receiving the target task corresponding to the task scheduling request from the scheduling node includes: Read the task nature of the target task corresponding to the task scheduling request; In response to the fact that the task is an anti-affinity task and there are no other existing tasks belonging to the anti-affinity task currently being executed locally, the target task corresponding to the task scheduling request is received from the scheduling node.

7. The method of claim 1, wherein, After collecting the storage load, the resource storage tool stores the storage load in shared storage maintained by at least two computing nodes. Reading the storage load of the target storage used locally includes: Read the storage load of the target storage used locally from the shared storage.

8. An apparatus for processing a task, characterized by include: The load reading module is configured to read the storage load of the target storage used locally in response to receiving a task scheduling request sent by the scheduling node, wherein the storage load is collected by a resource collection tool running locally; The task receiving module is configured to receive the target task corresponding to the task scheduling request from the scheduling node in response to the storage load being less than or equal to a first load threshold. The task processing module is configured to process the target task locally using the target storage.

9. An electronic device, the electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

10. A computer-readable medium having stored thereon computer program instructions that can be executed by a processor to implement the method as described in any one of claims 1 to 7.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images