Automatic task processing method and device

Abstract

The invention discloses an automatic task processing method and device, and the method comprises the steps: receiving a to-be-processed task uploaded by each user, and generating a plurality of task queues based on the to-be-processed task; determining estimated resource occupation information of the to-be-processed task, and adjusting a current node cluster based on the estimated resource occupation information to obtain a target node cluster; distributing the to-be-processed tasks in the task queue to each node in the target node cluster according to a predetermined sequence; and after each node finishes processing the to-be-processed task, returning a task processing result to a user, and dynamically optimizing the distribution sequence of the to-be-processed task and the node scale of the current node cluster by comparing the processing capabilities of the to-be-processed task and the current node cluster so as to improve the efficiency of automatic task processing.

Description

Technical Field

[0001] The present application relates to the technical field of task processing, and more specifically, to an automated task processing method and device. Background Art

[0002] In the field of modern enterprise applications and data processing, automated task processing and resource scheduling face significant challenges. As the business and data volume of enterprises grow rapidly, the complexity and number of tasks also increase dramatically, involving data processing, analysis, report generation, system maintenance and other background services. To meet these challenges, current solutions include the use of distributed computing frameworks such as Apache Hadoop and Apache Spark, as well as task scheduling systems such as Apache Airflow and Kubernetes. These systems can execute tasks in a distributed manner on multiple machines, aiming to ensure data integrity and effective parallel processing of tasks.

[0003] Although these existing technologies provide a certain degree of automation and efficiency, they still face problems such as configuration complexity, insufficient resource allocation, and lack of flexibility, especially in a dynamically changing business environment. In addition, existing systems often require professional technical knowledge to manage effectively, which may be a barrier for many enterprises. Therefore, there is a demand for more intelligent, adaptive and easy-to-manage automated task processing and resource scheduling solutions to better meet the needs of modern enterprises in a rapidly changing market environment. Summary of the invention

[0004] The present invention provides an automated task processing method and device to solve the technical problem that the task processing automation lacks flexibility in the prior art, resulting in low efficiency of task automation processing. The method comprises:

[0005] Receive the pending tasks uploaded by each user, and generate multiple task queues based on the pending tasks;

[0006] Determine the estimated resource occupancy information of the task to be processed, and adjust the current node cluster based on the estimated resource occupancy information to obtain a target node cluster;

[0007] Dispatching the pending tasks in the task queue to each node in the target node cluster in a predetermined order;

[0008] After each node completes processing the pending tasks, the task processing result is returned to the user.

[0009] In some specific embodiments, the receiving of the pending tasks uploaded by each user and the generation of multiple task queues based on the pending tasks are specifically as follows:

[0010] In response to a task instruction sent by a user, sending the to-be-processed task to a coordination manager through an API port;

[0011] Based on the tasks to be processed, the multiple task queues are generated by the coordination manager.

[0012] In some specific embodiments, the estimated resource occupancy information of the task to be processed is determined, and the current node cluster is adjusted based on the estimated resource occupancy information to obtain the target node cluster, specifically:

[0013] Determine the sub-estimated resource occupancy information corresponding to each task queue, and sum up each of the sub-estimated resource occupancy information to obtain the estimated resource occupancy information of the task to be processed;

[0014] Comparing the estimated resource occupancy information with the available resource information of the current node cluster to obtain a resource comparison result;

[0015] The current node cluster is adjusted based on the resource comparison result to obtain a target node cluster.

[0016] In some specific embodiments, the estimated resource occupancy information is compared with the available resource information of the current node cluster to obtain a resource comparison result, specifically:

[0017] Obtain node information uploaded by the current node cluster according to a predetermined collection frequency, wherein the node information includes the number of management processes, node CPU usage, node memory usage, and node network data usage;

[0018] Determine available resource information of the current node cluster based on the node information;

[0019] Comparing the estimated resource occupancy information with the available resource information of the current node cluster based on a first preset formula to obtain a resource estimation difference, and using the resource estimation difference as the resource comparison result;

[0020] Wherein, the first preset formula is:

[0021] R ResTotal =∑R NodeResUnused -∑R TaskResRequired ;

[0022] Among them, ∑R NodeResUnused is the available resource information of the current node cluster, ∑R TaskResRequired is the estimated resource occupancy information, R ResTotal A difference is estimated for the resource.

[0023] In some specific embodiments, the current node cluster is adjusted based on the resource comparison result to obtain a target node cluster, specifically:

[0024] When the resource estimation difference exceeds a first preset value, the current node cluster is in a state of excess resources, and nodes in the current node cluster whose load utilization is lower than a preset utilization are recycled to obtain the target node cluster;

[0025] When the resource estimation difference is lower than a second preset value, the current node cluster is in a state of insufficient resources, the number of nodes to be added is determined based on a second preset formula, and nodes of the number of nodes to be added are added to the current node cluster to obtain the target node cluster;

[0026] Wherein, the second preset formula is:

[0027]

[0028] Wherein, N is the number of nodes to be added, N NodeLimit is the maximum number of nodes, R NodeUnit Refers to the number of current node type groups.

[0029] In some specific embodiments, the tasks to be processed in the task queue are dispatched to each node in the target node cluster in a predetermined order, specifically:

[0030] Sorting the task sequence according to the order of task upload time by the coordination manager;

[0031] Sort the target node cluster by load utilization rate of each node from low to high;

[0032] The coordination manager distributes the tasks in the task sequence to the target node cluster in the order, and distributes the tasks in the task sequence to each node in descending order according to the load utilization of each node.

[0033] In some specific embodiments, before the coordination manager distributes the tasks in the task sequence to each node according to the load utilization rate of each node from high to low, the process further includes:

[0034] Sending registration information to the coordination manager through the execution machine in each node, and controlling the execution machine to successfully register in the coordination manager through the registration information;

[0035] Heartbeat detection is performed on the coordination manager and each execution machine at a predetermined sending frequency to determine the connection status between the coordination manager and each execution machine.

[0036] Accordingly, the present invention also proposes an automated task processing device, the device comprising:

[0037] A receiving module, used for receiving the pending tasks uploaded by each user, and generating multiple task queues based on the pending tasks;

[0038] A node adjustment module, used to determine the estimated resource occupancy information of the task to be processed, and adjust the current node cluster based on the estimated resource occupancy information to obtain a target node cluster;

[0039] A task dispatching module, used to dispatch the pending tasks in the task queue to each node in the target node cluster according to a predetermined order;

[0040] The result feedback module is used to return the task processing result to the user after each node completes processing the pending task.

[0041] Accordingly, the present invention further proposes a computing device, comprising:

[0042] Memory and processor;

[0043] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of any one of the above automated task processing methods are implemented.

[0044] Accordingly, the present invention further proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, can implement the steps of any one of the above-described automated task processing methods.

[0045] By applying the above technical scheme, an automated task processing method is proposed, the method comprising: receiving pending tasks uploaded by each user, and generating multiple task queues based on the pending tasks; determining the estimated resource occupancy information of the pending tasks, and adjusting the current node cluster based on the estimated resource occupancy information to obtain a target node cluster; dispatching the pending tasks in the task queue to each node in the target node cluster in a predetermined order; after each node completes processing the pending tasks, returning the task processing result to the user, dynamically adjusting the node cluster according to the pending tasks, and dynamically allocating the pending tasks to the node cluster, thereby realizing full automation of task processing and improving the efficiency of automated task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 It is a flowchart of an automated task processing method provided in an embodiment of the present application;

[0048] Figure 2 It is a structural diagram of each component in an automated task processing method provided in an embodiment of the present application;

[0049] Figure 3 It is a flowchart of task dispatching in an automated task processing method provided in an embodiment of the present application;

[0050] Figure 4 It is a structural schematic diagram of an automated task processing device provided in an embodiment of the present application;

[0051] Figure 5 is another flowchart of an automated task processing method provided by an embodiment of the present application;

[0052] Figure 6 It is a structural block diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0054] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0055] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0056] First, the terms involved in one or more embodiments of this specification are explained.

[0057] 1. GRPC / mux: gRPC multiplexing is a technology used for gRPC communication that allows multiple gRPC requests to be transmitted in parallel on a single TCP connection, improving communication efficiency.

[0058] 2. Relay: Relay refers to a device, system or service used to transfer data, information or signals so that they can be transmitted from one place to another.

[0059] 3. Reylet: The execution machine that actually executes the tasks issued.

[0060] 4. Heartbeat: A heartbeat usually refers to a small data packet or signal sent periodically to detect whether a device, service, or connection is still active or available. In computer networks, heartbeats are often used to monitor the health of network devices.

[0061] 5. Round-robin scheduling: Round-robin scheduling is a task scheduling algorithm in which tasks are assigned to available execution units (such as processors or servers) in sequence to ensure fair distribution of workload regardless of resource load.

[0062] like Figure 1 As shown, the present application proposes an automated task processing method, the method comprising the following steps:

[0063] Step S101, receiving the pending tasks uploaded by each user, and generating multiple task queues based on the pending tasks.

[0064] In this embodiment, the user uploads the tasks to be processed through the terminal device, which can be a computer, mobile phone, etc. The uploaded tasks are divided into multiple task queues as tasks to be processed, wherein each task queue includes at least one task, and the task queue division rule is to divide tasks of the same task type or of a competitive nature into the same task queue, and at the same time sort them in the same task queue according to the time order in which the tasks are received.

[0065] In some embodiments of the present application, the receiving of the pending tasks uploaded by each user and the generation of multiple task queues based on the pending tasks are specifically:

[0066] In response to a task instruction sent by a user, sending the to-be-processed task to a coordination manager through an API port;

[0067] Based on the tasks to be processed, the multiple task queues are generated by the coordination manager.

[0068] In this embodiment, Figure 2 , Figure 3 and Figure 5 As shown, in the intranet, users submit pending tasks, which are transmitted to the coordinator through the API Server. APIs are predefined functions that aim to provide applications and developers with the ability to access a set of routines based on certain software or hardware without having to access the source code or understand the details of the internal working mechanism. The pending tasks can be sent to the coordinator through the API port. The role of the coordinator in this solution is to serve as the data interaction between the node cluster and the user, forwarding the data sent by the user in the intranet to the node cluster in the external network, and sending the information returned by the node cluster back to the user terminal.

[0069] In this embodiment, Figure 3 As shown in the figure, after the user submits the pending tasks, the coordination manager divides the pending tasks into multiple task queues, and waits for the coordination manager to take the pending tasks in the task queues and then send them to the node cluster. Furthermore, the current node cluster can also be divided into multiple node task pools according to actual needs, or divided into multiple regional node clusters according to different regions, for example Figure 2 In the example, the Beijing node cluster, the Shanghai node cluster or the Hangzhou node cluster can process different pending tasks in a targeted manner through the above distinction or division, thereby further improving the efficiency of task automation processing.

[0070] Step S102: determining the estimated resource occupancy information of the task to be processed, and adjusting the current node cluster based on the estimated resource occupancy information to obtain a target node cluster.

[0071] In this embodiment, the estimated resource occupancy information refers to the estimated value of the node resources occupied by the task. For example, the estimated resource occupancy information of a task uploaded by a user is 20% of the first node, which means that the first node needs to occupy 20% of its processing capacity when processing the task. The first node can process up to 5 of the above tasks. By determining the estimated resource occupancy information, the number of nodes required for the task to be processed can be confirmed, and then it can be judged whether the current node cluster needs to be adjusted to meet the needs of the current task to be processed, and the target node cluster can be obtained after adjustment.

[0072] In order to adjust the current node cluster to meet the node requirements of the task to be processed, in some embodiments of the present application, the estimated resource occupancy information of the task to be processed is determined, and the current node cluster is adjusted based on the estimated resource occupancy information to obtain the target node cluster, specifically:

[0073] Determine the sub-estimated resource occupancy information corresponding to each task queue, and sum up each of the sub-estimated resource occupancy information to obtain the estimated resource occupancy information of the task to be processed;

[0074] Comparing the estimated resource occupancy information with the available resource information of the current node cluster to obtain a resource comparison result;

[0075] The current node cluster is adjusted based on the resource comparison result to obtain a target node cluster.

[0076] In this embodiment, the sub-estimated resource occupancy information corresponding to each task queue is calculated respectively, and all the sub-estimated resource occupancy information is added up to obtain the estimated resource occupancy information of all tasks to be processed, and at the same time, the available resource information of the current node cluster is obtained, and the two are compared to obtain a resource comparison result. According to the resource comparison result, the current node cluster is adjusted so that the node processing capacity of the current node cluster can meet the needs of the tasks to be processed. Therefore, in theory, the available resource information should be greater than or equal to the estimated resource occupancy information.

[0077] In order to obtain a resource comparison result and adjust the current node cluster according to the resource comparison result, in some embodiments of the present application, the estimated resource occupancy information is compared with the available resource information of the current node cluster to obtain a resource comparison result, specifically:

[0078] Obtain node information uploaded by the current node cluster according to a predetermined collection frequency, wherein the node information includes the number of management processes, node CPU usage, node memory usage, and node network data usage;

[0079] Determine available resource information of the current node cluster based on the node information;

[0080] Comparing the estimated resource occupancy information with the available resource information of the current node cluster based on a first preset formula to obtain a resource estimation difference, and using the resource estimation difference as the resource comparison result;

[0081] Wherein, the first preset formula is:

[0082] R ResTotal =∑R NodeResUnused -∑R TaskResRequired ;

[0083] Among them, ∑R NodeResUnused is the available resource information of the current node cluster, ∑R TaskResRequired is the estimated resource occupancy information, R ResTotal A difference is estimated for the resource.

[0084] In this embodiment, the node information includes but is not limited to the number of management processes, node CPU usage, node memory usage, and node network data usage mentioned above. At the same time, the node reports the node information according to a predetermined collection frequency. In this embodiment, a resource management module is provided to save and manage the above node information.

[0085] In this embodiment, the available resource information of the current node cluster is obtained based on the node information sent by each node in the node cluster, and the difference between the estimated resource occupancy information and the available resource information of the current node cluster is calculated by a first preset formula as the resource estimation difference. The current node cluster is adjusted according to the resource estimation difference to obtain the target node cluster.

[0086] In order to adjust the current node cluster according to the resource comparison result, in some embodiments of the present application, the current node cluster is adjusted based on the resource comparison result to obtain the target node cluster, specifically:

[0087] When the resource estimation difference exceeds a first preset value, the current node cluster is in a state of excess resources, and nodes in the current node cluster whose load utilization is lower than a preset utilization are recycled to obtain the target node cluster;

[0088] When the resource estimation difference is lower than a second preset value, the current node cluster is in a state of insufficient resources, the number of nodes to be added is determined based on a second preset formula, and nodes of the number of nodes to be added are added to the current node cluster to obtain the target node cluster;

[0089] Wherein, the second preset formula is:

[0090]

[0091] Wherein, N is the number of nodes to be added, N NodeLimit for;

[0092] In this embodiment, when the resource estimation difference exceeds the first preset value, it means that the current node cluster can meet the processing needs of the task to be processed, but some nodes in the current node cluster are in a low working state, that is, the number of nodes in the current node cluster is oversaturated, and it is necessary to recycle the nodes whose processing efficiency or load utilization is lower than the preset utilization to avoid wasting node resources. In this embodiment, the preset utilization can be set to 20%. When the load utilization of a node is lower than 20%, the node will be recycled.

[0093] In this embodiment, when the resource estimation difference is lower than the second preset value, it means that the current node cluster is not sufficient to handle so many tasks to be processed, and new nodes need to be added to improve the processing capacity of the current node cluster. The number of new nodes is obtained by the second preset formula.

[0094] By making comprehensive judgments based on pending tasks and the current status of the node cluster, we can adapt to changing task and resource requirements and continuously optimize the node scale of the node cluster, thereby better meeting the needs of enterprises in a rapidly changing market environment.

[0095] Step S103: dispatching the pending tasks in the task queue to each node in the target node cluster according to a predetermined order.

[0096] In this embodiment, Figure 3 As shown, after the adjustment of the current node cluster is completed, the ideal target node cluster is obtained, and the pending tasks in the task queue are dispatched in a predetermined order so that each node in the target node cluster receives the pending tasks.

[0097] In order to achieve orderly dispatch of tasks to be processed, in some embodiments of the present application, the tasks to be processed in the task queue are dispatched to each node in the target node cluster in a predetermined order, specifically:

[0098] Sorting the task sequence according to the order of task upload time by the coordination manager;

[0099] Sort the target node cluster by load utilization rate of each node from low to high;

[0100] The coordination manager distributes the tasks in the task sequence to the target node cluster in the order, and distributes the tasks in the task sequence to each node in descending order according to the load utilization of each node.

[0101] In this embodiment, on the task sequence side, this solution uses a round-robin scheduling method to dispatch tasks. Round-robin scheduling is a task scheduling algorithm in which tasks are sequentially assigned to available execution units (such as processors or servers) to ensure fair distribution of workloads regardless of resource loads. That is, the task sequence is sorted in the order of task upload time, and the tasks to be processed that were uploaded first are preferentially dispatched. At the same time, in order to further optimize the automated processing process and the processing efficiency of each node, on the node cluster side, the dispatched tasks are also preferentially sent to nodes with low load utilization to avoid excessively high or low load utilization of a certain node.

[0102] In order to ensure that the node can normally process the pending tasks, in some embodiments of the present application, before the coordination manager distributes the tasks in the task sequence to each node according to the load utilization rate of each node from high to low, it also includes:

[0103] Sending registration information to the coordination manager through the execution machine in each node, and controlling the execution machine to successfully register in the coordination manager through the registration information;

[0104] Heartbeat detection is performed on the coordination manager and each execution machine at a predetermined sending frequency to determine the connection status between the coordination manager and each execution machine.

[0105] In this embodiment, Figure 5 As shown in the figure, before processing a task, each node needs to register with the coordination manager. After successful registration, the heartbeat is used to detect whether the connection between the two parties is normal and whether tasks or data can be sent and received normally. The heartbeat usually refers to a small data packet or signal sent periodically to detect whether the device, service or connection is still active or available. In computer networks, heartbeats are usually used to monitor the health of network devices. At the same time, Figure 5 As shown, when the node is processing the pending task, it is also necessary to perform heartbeat detection according to the predetermined sending frequency to ensure that the connection in the task processing stage is normal.

[0106] Step S104: After each node completes processing the pending task, a task processing result is returned to the user.

[0107] By applying the above technical solution, an automated task processing method is proposed, the method comprising: receiving pending tasks uploaded by each user, and generating multiple task queues based on the pending tasks; determining the estimated resource occupancy information of the pending tasks, and adjusting the current node cluster based on the estimated resource occupancy information to obtain a target node cluster; dispatching the pending tasks in the task queue to each node in the target node cluster in a predetermined order; after each node completes processing the pending tasks, returning the task processing result to the user, and dynamically optimizing the dispatching order of the pending tasks and the node scale of the current node cluster by comparing the processing capabilities of the pending tasks with those of the current node cluster, so as to improve the efficiency of automated task processing.

[0108] The present application also proposes an automated task processing device, such as Figure 4 As shown, the device comprises:

[0109] The receiving module 401 is used to receive the pending tasks uploaded by each user, and generate multiple task queues based on the pending tasks;

[0110] The node adjustment module 402 is used to determine the estimated resource occupation information of the task to be processed, and adjust the current node cluster based on the estimated resource occupation information to obtain a target node cluster;

[0111] The task dispatching module 403 is used to dispatch the pending tasks in the task queue to each node in the target node cluster according to a predetermined order;

[0112] The result feedback module 404 is used to return the task processing result to the user after each node completes processing the task to be processed.

[0113] Figure 6 The structure block diagram of a computing device 500 provided according to an embodiment of the present specification is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.

[0114] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0115] In one embodiment of the present specification, the above components of the computing device 500 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0116] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 500 may also be a mobile or stationary server.

[0117] The processor 520 is used to execute the following computer executable instructions, which implement the steps of the above power management method when executed by the processor. The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the above power management method belong to the same concept, and the details of the technical solution of the computing device that are not described in detail can be found in the description of the technical solution of the above power management method.

[0118] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above power management method when executed by a processor.

[0119] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the power management method described above are of the same concept, and the details not described in detail in the technical scheme of the storage medium can be found in the description of the technical scheme of the power management method described above.

[0120] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above power management method.

[0121] The above is an illustrative solution of a computer program of this embodiment. It should be noted that the technical solution of the computer program and the technical solution of the power management method described above are of the same concept, and the details not described in detail in the technical solution of the computer program can be found in the description of the technical solution of the power management method described above.

[0122] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0123] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0124] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0125] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. An automated task processing method, characterized in that: The method comprises: Receive the pending tasks uploaded by each user, and generate multiple task queues based on the pending tasks; Determine the estimated resource occupancy information of the task to be processed, and adjust the current node cluster based on the estimated resource occupancy information to obtain a target node cluster; Dispatching the pending tasks in the task queue to each node in the target node cluster in a predetermined order; After each node completes processing the pending tasks, the task processing result is returned to the user.

2. The method according to claim 1, characterized in that The receiving of the pending tasks uploaded by each user and generating multiple task queues based on the pending tasks are specifically as follows: In response to a task instruction sent by a user, sending the to-be-processed task to a coordination manager through an API port; Based on the tasks to be processed, the multiple task queues are generated by the coordination manager.

3. The method according to claim 2, characterized in that Determine the estimated resource occupancy information of the task to be processed, and adjust the current node cluster based on the estimated resource occupancy information to obtain a target node cluster, specifically: Determine the sub-estimated resource occupancy information corresponding to each task queue, and sum up each of the sub-estimated resource occupancy information to obtain the estimated resource occupancy information of the task to be processed; Comparing the estimated resource occupancy information with the available resource information of the current node cluster to obtain a resource comparison result; The current node cluster is adjusted based on the resource comparison result to obtain a target node cluster.

4. The method according to claim 3, characterized in that The estimated resource occupancy information is compared with the available resource information of the current node cluster to obtain a resource comparison result, specifically: Obtain node information uploaded by the current node cluster according to a predetermined collection frequency, wherein the node information includes the number of management processes, node CPU usage, node memory usage, and node network data usage; Determine available resource information of the current node cluster based on the node information; Comparing the estimated resource occupancy information with the available resource information of the current node cluster based on a first preset formula to obtain a resource estimation difference, and using the resource estimation difference as the resource comparison result; Wherein, the first preset formula is: R ResTotal =∑R NodeResUnused -∑R TaskResRequired ; Among them, ∑R NodeResUnused is the available resource information of the current node cluster, ∑R TaskResRequired is the estimated resource occupancy information, R ResTotal A difference is estimated for the resource.

5. The method according to claim 4, characterized in that The current node cluster is adjusted based on the resource comparison result to obtain a target node cluster, specifically: When the resource estimation difference exceeds a first preset value, the current node cluster is in a state of excess resources, and nodes in the current node cluster whose load utilization is lower than a preset utilization are recycled to obtain the target node cluster; When the resource estimation difference is lower than a second preset value, the current node cluster is in a state of insufficient resources, the number of nodes to be added is determined based on a second preset formula, and nodes of the number of nodes to be added are added to the current node cluster to obtain the target node cluster; Wherein, the second preset formula is: Wherein, N is the number of nodes to be added, N NodeLimit is the maximum number of nodes, R NodeUnit Refers to the number of current node type groups.

6. The method according to claim 2, characterized in that The dispatching of the pending tasks in the task queue to each node in the target node cluster according to a predetermined order is specifically as follows: Sorting the task sequence according to the order of task upload time by the coordination manager; Sort the target node cluster by load utilization rate of each node from low to high; The coordination manager distributes the tasks in the task sequence to the target node cluster in the order, and distributes the tasks in the task sequence to each node in descending order according to the load utilization of each node.

7. The method according to claim 1, characterized in that Before distributing the tasks in the task sequence to each node according to the load utilization rate of each node from high to low by the coordination manager, the method further includes: Sending registration information to the coordination manager through the execution machine in each node, and controlling the execution machine to successfully register in the coordination manager through the registration information; Heartbeat detection is performed on the coordination manager and each execution machine at a predetermined sending frequency to determine the connection status between the coordination manager and each execution machine.

8. An automated task processing device, characterized in that: The device comprises: A receiving module, used for receiving the pending tasks uploaded by each user, and generating multiple task queues based on the pending tasks; A node adjustment module, used to determine the estimated resource occupancy information of the task to be processed, and adjust the current node cluster based on the estimated resource occupancy information to obtain a target node cluster; A task dispatching module, used to dispatch the pending tasks in the task queue to each node in the target node cluster according to a predetermined order; The result feedback module is used to return the task processing result to the user after each node completes processing the pending task.

9. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the automated task processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the automated task processing method according to any one of claims 1 to 7.

Images