Distributed task push method and device, electronic device, and storage medium

Abstract

The present invention provides a distributed task push method and device, electronic device, and storage medium, which are applied to a distributed task push system, the system comprising multiple master nodes and multiple task execution nodes, each master node managing at least one task execution node, each task execution node executing one or more task instances of a task, obtaining one or more task instances of each of the multiple tasks to be pushed; determining a task push strategy for distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node; and distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task push strategy. The solution provided by the present invention solves the technical problem of low efficiency in the process of executing concurrent tasks in the existing distributed scheduling system.

Description

Technical Field

[0001] The present invention relates to the field of big data processing, and in particular to a distributed task pushing method and device, electronic equipment, and storage medium. Background Art

[0002] Distributed scheduling system refers to the control of executing a large number of tasks through timers or one-time control, using multi-threading and multi-host concurrently to complete a large number of tasks, while executing dependent tasks in order and controlling the status update. It is widely used in the construction of big data warehouses.

[0003] At present, the most widely used distributed scheduling system in the industry is the Apache open source Dolphin Scheduler. Then, in the process of executing distributed task scheduling, the master (main process) starts a thread to submit a task, which needs to obtain a task from the database. After submission, it needs to continuously poll and wait, occupying resources. If there are 100,000 tasks that need to run at the same time, because the data structure is divided into three layers, namely definition, instance and task instance, it is necessary to set up a 200,000 thread pool and 300,000 threads to continuously poll the status and update the operation in real time. However, the polling operation is to query multiple tables of the database multiple times, and the database query pressure is also relatively large. If the master specifies the number of threads, the throughput will be greatly reduced, and the threads are waiting for polling to update the status; when the worker (member process) obtains the task, there will be a zk lock, and only one worker can obtain this lock. After obtaining the lock, it obtains the task and performs a series of operations, such as database query, data confirmation, local folder and file operation, task removal, and then releases the lock. This process will take a long time, and other workers can only wait, which greatly reduces the efficiency of concurrency.

[0004] Therefore, the existing distributed scheduling system has the technical problem of low efficiency in executing concurrent tasks. Summary of the invention

[0005] In view of the above problems, the present invention proposes a distributed task pushing method and device, an electronic device, and a storage medium to at least solve the technical problem of low efficiency in the process of executing concurrent tasks in the existing distributed scheduling system.

[0006] According to a first aspect of the present invention, a distributed task pushing method is provided, which is applied to a distributed task pushing system, wherein the system comprises a plurality of master nodes and a plurality of task execution nodes, each master node managing at least one task execution node, and each task execution node executing one or more task instances of a task, the method comprising: obtaining one or more task instances of each of a plurality of tasks to be pushed; determining a task pushing strategy for distributing the one or more task instances of each of the plurality of tasks to be pushed to the task execution nodes managed by each master node; wherein the task pushing strategy is formulated by first quantity information of the plurality of master nodes, second quantity information of the plurality of task execution nodes, and one or more task instances of each of the plurality of tasks to be pushed; and distributing the one or more task instances of each of the plurality of tasks to be pushed to the task execution nodes managed by each master node according to the task pushing strategy.

[0007] Optionally, determining a task push strategy for distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node includes: selecting any master node among the multiple master nodes as the first master node, broadcasting the master node ID corresponding to the first master node to the remaining master nodes among the multiple master nodes except the first master node; filtering out the second master node corresponding to the smallest or largest master node ID through any master node, and using the second master node as the leader node; detecting the estimated execution time corresponding to the execution of the multiple tasks through the leader node; if the estimated execution time does not exceed a preset value, selecting the first task push strategy among the task push strategies; if the estimated execution time exceeds the preset value, selecting the second task push strategy among the task push strategies, wherein the second task push strategy is formulated by the first quantity information of the multiple master nodes, the health factor corresponding to each master node, the second quantity information of the multiple task execution nodes, the one or more task instances of each of the multiple tasks to be pushed, and the health factor corresponding to each task execution node.

[0008] Optionally, distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task pushing strategy includes: grouping the multiple tasks according to a first grouping algorithm corresponding to the first task pushing strategy; pushing the multiple tasks to each master node in sequence according to the grouping results corresponding to the first grouping algorithm; selecting any master node among the multiple master nodes as the third master node, and pushing one or more task instances corresponding to the tasks on the third master node to each task execution node managed by the third master node according to the first grouping algorithm; wherein the first grouping algorithm is to perform a modulo operation on the task ID corresponding to each task and the first quantity information, or to perform a modulo operation on the task instance ID corresponding to each task instance and the second quantity information.

[0009] Optionally, distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task push strategy includes: grouping the multiple tasks according to a second grouping algorithm corresponding to the second task push strategy; pushing the multiple tasks to each master node in sequence according to the grouping results corresponding to the second grouping algorithm; selecting any one of the multiple master nodes as the fourth master node, and pushing one or more task instances corresponding to the tasks on the fourth master node to each task execution node managed by the fourth master node according to the second grouping algorithm; wherein the second grouping algorithm is to perform a modulo operation on the task ID corresponding to each task and the health factor corresponding to each master node; or to perform a modulo operation on the task instance ID corresponding to each task instance and the health factor corresponding to each task execution node.

[0010] Optionally, after distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task push strategy, the method further includes: refreshing the health factors corresponding to each master node and each task execution node.

[0011] Optionally, after distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task push strategy, the method also includes: after each task execution node executes each task instance, receiving log information and execution results corresponding to the execution of each task instance fed back by each task execution node.

[0012] Optionally, after distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task push strategy, the method further includes: sending a query request to each master node to query the log information and execution progress information of each task instance executed by each task execution node, and each master node sending the query request to each task execution node; receiving the log information and execution progress information fed back by each task execution node according to the query request.

[0013] According to the second aspect of the present invention, a distributed task pushing device is also provided, which is applied to a distributed task pushing system, wherein the system includes multiple master nodes and multiple task execution nodes, each master node manages at least one task execution node, and each task execution node executes one or more task instances of the task, and the device includes: an acquisition module, used to acquire one or more task instances of each of the multiple tasks to be pushed; a determination module, used to determine a task pushing strategy for distributing the one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node; wherein the task pushing strategy is formulated by the first quantity information of the multiple master nodes, the second quantity information of the multiple task execution nodes, and the one or more task instances of each of the multiple tasks to be pushed; a distribution module, used to distribute the one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task push strategy.

[0014] Optionally, the determination module includes: a broadcast unit, used to select any master node among the multiple master nodes as the first master node, and broadcast the master node ID corresponding to the first master node to the remaining master nodes among the multiple master nodes except the first master node; a determination unit, used to screen out the second master node corresponding to the smallest or largest master node ID through any master node, and use the second master node as the leader node; a detection unit, used to detect the estimated execution time corresponding to the execution of the multiple tasks through the leader node; a selection unit, used to select the first task push strategy among the task push strategies when the estimated execution time does not exceed a preset value; when the estimated execution time exceeds the preset value, select the second task push strategy among the task push strategies, wherein the second task push strategy is formulated by the first quantity information of the multiple master nodes, the health factor corresponding to each master node, the second quantity information of the multiple task execution nodes, one or more task instances of each of the multiple tasks to be pushed, and the health factor corresponding to each task execution node.

[0015] Optionally, the distribution module includes: a first grouping unit, used to group the multiple tasks according to a first grouping algorithm corresponding to the first task pushing strategy; a first pushing unit, used to push the multiple tasks to each master node in sequence according to the grouping results corresponding to the first grouping algorithm; a second pushing unit, used to select any one of the multiple master nodes as the third master node, and push one or more task instances corresponding to the tasks on the third master node to each task execution node managed by the third master node according to the first grouping algorithm; wherein the first grouping algorithm is to perform a modulo operation on the task ID corresponding to each task and the first quantity information, or to perform a modulo operation on the task instance ID corresponding to each task instance and the second quantity information.

[0016] Optionally, the distribution module includes: a second grouping unit, used to group the multiple tasks according to a second grouping algorithm corresponding to the second task push strategy; a third pushing unit, used to push the multiple tasks to each master node in sequence according to the grouping results corresponding to the second grouping algorithm; a fourth pushing unit, used to select any one of the multiple master nodes as the fourth master node, and push one or more task instances corresponding to the tasks on the fourth master node to each task execution node managed by the fourth master node according to the second grouping algorithm; wherein the second grouping algorithm is to perform a modulo operation on the task ID corresponding to each task and the health factor corresponding to each master node; or to perform a modulo operation on the task instance ID corresponding to each task instance and the health factor corresponding to each task execution node.

[0017] Optionally, the device also includes: a refresh module, which is used to refresh the health factors corresponding to each master node and each task execution node after the distribution module distributes one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task push strategy.

[0018] Optionally, the device also includes: a first receiving module, which is used to receive log information and execution results corresponding to the execution of each task instance fed back by each task execution node after the distribution module distributes one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task push strategy, and after each task execution node executes each task instance.

[0019] Optionally, the device also includes: a sending module, which is used to send a query request to each master node to query the log information and execution progress information of each task instance executed by each task execution node after the distribution module distributes one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task push strategy, and each master node sends the query request to each task execution node; a second receiving module, which is used to receive the log information and execution progress information fed back by each task execution node according to the query request.

[0020] According to a third aspect of the present invention, there is further provided an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0021] According to a fourth aspect of the present invention, there is further provided a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned device embodiments when running.

[0022] The technical solution provided by the present invention is applied to a distributed task pushing system. The system includes multiple master nodes and multiple task execution nodes. Each master node manages at least one task execution node. Each task execution node executes one or more task instances of a task. First, one or more task instances of each of the multiple tasks to be pushed are obtained; then, a task pushing strategy for distributing the one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node is determined. The task pushing strategy is formulated by first quantity information of the multiple master nodes, second quantity information of the multiple task execution nodes, and one or more task instances of each of the multiple tasks to be pushed; according to the task pushing strategy, one or more task instances of each of the multiple tasks to be pushed are distributed to the task execution nodes managed by each master node, thereby achieving the effect of efficiently pushing distributed tasks. By distributing multiple tasks to multiple task execution nodes, the task progress query work can be distributed to each master node for operation, thereby improving the query efficiency of the task progress, and solving the technical problem of low efficiency in the process of executing concurrent tasks in the existing distributed scheduling system.

[0023] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below.

[0024] Based on the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will become more aware of the above and other objects, advantages and features of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0026] Figure 1 is a flow chart of a distributed task pushing method provided according to an embodiment of the present invention;

[0027] Figure 2 It is a schematic diagram of an architecture of a distributed scheduling system based on Akka provided according to an embodiment of the present invention;

[0028] Figure 3 A workflow diagram of a distributed scheduling system based on Akka is provided according to an embodiment of the present invention;

[0029] Figure 4 is a structural block diagram of a distributed task pushing device provided according to an embodiment of the present invention;

[0030] Figure 5 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such usage is interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the term "including" and its variants are to be interpreted as open-ended terms meaning "including but not limited to".

[0033] In order to solve the technical problems existing in the related art, a distributed task push method is provided in this embodiment. The technical scheme of the present invention and how the technical scheme of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0034] Figure 1 is a flow chart of a distributed task push method provided according to an embodiment of the present invention. Figure 1 As shown, the method is applied to a distributed task push system, which includes multiple master nodes and multiple task execution nodes. Each master node manages at least one task execution node. Each task execution node executes one or more task instances of a task. The process includes the following steps:

[0035] Step S102, obtaining one or more task instances of each of the multiple tasks to be pushed;

[0036] In an application scenario of the present embodiment, the Master (main process)-Worker (member process) mode is adopted, which is a parallel mode. The Master is the main process (i.e., the above-mentioned main node). A Worker process queue is maintained in the Master. After receiving a task, the Master splits the task and splits a large and complex batch business into several small businesses. As long as they do not affect each other, they can be divided and conquered and independent of each other, and assigned to each Worker, so that each Worker can run its own subtask. For example, through multi-threading, multi-process or even multi-machine joint computing, the split small businesses are handed over to more CPUs or machines for processing (i.e., the above-mentioned task execution nodes). Finally, the return results of the Workers are handed over to the Master for summary and finally returned to the initiator of the task.

[0037] In this embodiment, one or more task instances of each of the multiple tasks to be pushed are obtained so that the Master process and the Worker process can each share their respective tasks and collaboratively complete the information processing mode; the computing speed of the overall business is increased through concurrency / parallelism, and the system performance is squeezed to improve efficiency.

[0038] Step S104, determining a task push strategy for distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node; wherein the task push strategy is formulated by the first quantity information of the multiple master nodes, the second quantity information of the multiple task execution nodes, and the one or more task instances of each of the multiple tasks to be pushed;

[0039] The above-mentioned step S104 specifically includes: selecting any master node from multiple master nodes as the first master node, broadcasting the master node ID corresponding to the first master node to the remaining master nodes from the multiple master nodes except the first master node; screening out the second master node corresponding to the smallest or largest master node ID through any master node, and using the second master node as the leader node; detecting the estimated execution time corresponding to executing multiple tasks through the leader node; if the estimated execution time does not exceed the preset value, selecting the first task push strategy in the task push strategy; if the estimated execution time exceeds the preset value, selecting the second task push strategy in the task push strategy, wherein the second task push strategy is formulated according to the first quantity information of multiple master nodes, the health factor of each master node, the second quantity information of multiple task execution nodes, one or more task instances of each of the multiple tasks to be pushed, and the health factor of each task execution node.

[0040] In one example, each master broadcasts its own master ID to other masters; each master selects the master corresponding to the maximum master ID or the minimum master ID from all master IDs as the leadermaster (i.e., the above-mentioned leader node).

[0041] Furthermore, the leader master can detect the estimated execution time corresponding to the batch task. If the estimated execution time does not exceed the preset value, it means that the batch task is a simple task, and the first task push strategy can be used to complete the task push; if the estimated execution time does not exceed the preset value, it means that the batch task is a complex task, and the second task push strategy can be used. The difference between the first task push strategy and the second task push strategy is that the second task push strategy is formulated with a health factor corresponding to the master node or task execution node. The larger the health factor, the more tasks the master node or task execution node can execute.

[0042] Furthermore, the leader master can be used to determine the task ID, task information, grouping algorithm, timing time and other information; the leader master sends a preset message to all masters, and then all masters respond to the message, so as to count the number of surviving masters (i.e., the above-mentioned master nodes) (i.e., the above-mentioned first quantity information); the leader master can also be used to receive the task execution results, task execution progress information and task execution log information obtained by all masters, thereby realizing the task status query work of a large number of tasks being distributed to multiple masters through the leader master, greatly improving the query speed and efficiency, thereby querying the task execution status in real time.

[0043] In a possible implementation of this embodiment, multiple tasks are grouped according to a first grouping algorithm corresponding to a first task pushing strategy; the multiple tasks are pushed to each master node in turn according to the grouping result corresponding to the first grouping algorithm; any master node among the multiple master nodes is selected as a third master node, and one or more task instances corresponding to the tasks on the third master node are pushed to each task execution node managed by the third master node according to the first grouping algorithm; wherein the first grouping algorithm is to perform a modulo operation on the task ID and the first quantity information corresponding to each task, or to perform a modulo operation on the task instance ID and the second quantity information corresponding to each task instance.

[0044] In an example of this embodiment, the first grouping algorithm is represented by the following mathematical model:

[0045] mod(w,n);

[0046] Wherein, w is the task ID, and n represents the number of masters (ie, the first quantity information mentioned above); if w represents the task instance ID, then n represents the number of task execution nodes (ie, the second quantity information mentioned above).

[0047] The remainder is obtained by performing a modulo operation on the task ID value and the number of masters, and then multiple tasks are assigned to the corresponding masters in sequence according to the remainder, that is, modulo the number of groups; then, one or more task instances corresponding to each task on any master node are pushed to the corresponding task execution node according to the same algorithm grouping. This embodiment is suitable for the case where the execution time of batch tasks is short, that is, the first grouping algorithm is used to distribute the batch tasks for simple tasks, and then the same first grouping algorithm is used to distribute the one or more task instances of each task to the corresponding task execution node, so as to efficiently distribute the simple tasks.

[0048] In another possible implementation of the present embodiment, multiple tasks are grouped according to a second grouping algorithm corresponding to a second task push strategy; the multiple tasks are pushed to each master node in turn according to the grouping result corresponding to the second grouping algorithm; any one of the multiple master nodes is selected as a fourth master node, and one or more task instances corresponding to the tasks on the fourth master node are pushed to each task execution node managed by the fourth master node according to the second grouping algorithm; wherein the second grouping algorithm is to perform a modulo operation on the task ID corresponding to each task and the health factor corresponding to each master node; or to perform a modulo operation on the task instance ID corresponding to each task instance and the health factor corresponding to each task execution node.

[0049] In an example of this embodiment, the second grouping algorithm is represented by the following mathematical model:

[0050]

[0051] Among them, h i Indicates the health factor of the i-th master (values ​​range from 0 to n, 0 represents the first master). The health factor ranges from 1 to 10. The larger the value, the healthier the master. The healthier the master, the more tasks it can manage. For each master node, h i It can also be expressed as the health factor corresponding to the i-th task execution node (value range is 0-n, 0 represents the first task execution node). The health factor ranges from 1 to 10. The larger the value, the healthier the task execution node. The healthier the task execution node, the more task instances it can execute.

[0052] Based on the health factor and task ID value of each master, the second grouping algorithm is used to calculate the minimum value of j corresponding to each i. The minimum value of j is used as the master ID corresponding to each task, so as to group the batch tasks.

[0053] Furthermore, after distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task push strategy, it also includes: refreshing the health factors corresponding to each master node and each task execution node.

[0054] For example, after assigning tasks to each master or worker, the health of the master or worker is refreshed at a preset time (such as time m), thereby refreshing the remaining executable tasks of the master or worker in real time, so as to achieve real-time monitoring of the master or worker, making it easier for the leader master to more accurately grasp the current status of each master or each worker.

[0055] Step S106: Distribute one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task pushing strategy.

[0056] The distributed task pushing method provided by the embodiment of the present invention is applied to a distributed task pushing system, which includes multiple master nodes and multiple task execution nodes, each master node manages at least one task execution node, and each task execution node executes one or more task instances of a task. First, one or more task instances of each of the multiple tasks to be pushed are obtained; then, a task pushing strategy for distributing the one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node is determined, and the task pushing strategy is formulated by first quantity information of the multiple master nodes, second quantity information of the multiple task execution nodes, and one or more task instances of each of the multiple tasks to be pushed; according to the task pushing strategy, one or more task instances of each of the multiple tasks to be pushed are distributed to the task execution nodes managed by each master node, thereby achieving the effect of efficiently pushing distributed tasks, by distributing multiple tasks to multiple task execution nodes, and then being able to distribute task progress query work to each master node for operation, thereby improving the query efficiency of task progress, and solving the technical problem of low efficiency in the process of executing concurrent tasks in the existing distributed scheduling system.

[0057] Optionally, after the above step S106, it also includes: after each task execution node executes each task instance, receiving the log information and execution results corresponding to the execution of each task instance fed back by each task execution node. In this embodiment, after the task execution node completes the task, it actively feeds back the result of the task execution and the log information of the task execution to the corresponding master node, and then each master node feeds back to the leader node; wherein the result of the task execution includes at least: the heartbeat of the worker, the progress of the task execution (completed, not executed or failed to execute), instance status, machine status, election, etc. Figure 2 is a schematic diagram of a distributed scheduling system architecture based on Akka according to an embodiment of the present invention, such as Figure 2 As shown in the figure, each master can manage at least one worker. The leader master sends a query request to each master, which requests the master's heartbeat, task status, task instance status, worker machine status, task execution log information and election status, etc.; then the master sends the query request to the worker managed by the master, and then the worker responds to the query request and reports the execution progress information of the task instance, the worker's heartbeat, execution results, execution log and the worker's machine status to the master, and then feeds back to the leader master.

[0058] Optionally, after the above step S106, it also includes: sending a query request to each master node to query the log information and execution progress information of each task execution node executing each task instance, and each master node sends the query request to each task execution node; receiving the log information and execution progress information fed back by each task execution node according to the query request. In this embodiment, due to the situation that the task execution node fails to execute the task, the task execution node cannot feed back the task execution result in real time or accurately, so the leader master can actively send a query request to each master, and then each master sends the received query request to each task execution node managed by the master, and the task execution node feeds back the execution progress information and log information in real time during the task execution process.

[0059] The embodiment of the present invention is further described below in conjunction with a specific embodiment:

[0060] Figure 3 A workflow diagram of a distributed scheduling system based on Akka is provided according to an embodiment of the present invention. Figure 3 As shown, the following steps are included:

[0061] Step 1: The Master broadcasts and elects a leader (i.e., the above-mentioned leader node) (any member (worker) node may become a leader), and determines the number of groups n based on the broadcast surviving master (i.e., the above-mentioned master node);

[0062] Step 2: The leader master creates an update scheduling plan (i.e. the above-mentioned task scheduling strategy);

[0063] Step 3: Feedback the updated scheduling plan to the client;

[0064] Step 4. The Master Leader determines a grouping algorithm based on the number of groups n, defines the task ID as w, Algorithm 1 (the first grouping algorithm mentioned above), modulo the number of groups, mod(w, n) result is the number of groups, Algorithm 2 (the second grouping algorithm mentioned above), master health factor (1-10 points, the larger the number, the healthier), grouping: The minimum value of j in this formula is the grouping of tasks. The solution is that j can be traversed from 0 to n. The healthier the task, the more tasks it will be assigned to manage. The master health can be refreshed after a period of time m.

[0065] Step 5: Query task information and logs;

[0066] Step 6: Feedback the task information and logs to the client;

[0067] Step 7: Add the acquired scheduled tasks to the scheduling queue and generate task instances regularly;

[0068] Step 8: Assign workers to task instances based on worker health using the algorithm in step 4. Push the task instance to the corresponding worker via Akka and update the task instance status to assigned.

[0069] Step 9: Worker receives the task instance, adds it to the scheduling queue, and the worker thread pool starts executing the task and updates the task status;

[0070] Step 10: Feedback the task status to the master;

[0071] Step 11: After the worker completes the task, it pushes the status to the master. The master can actively request the task instance status to prevent the worker from not executing for a long time and triggering a retry.

[0072] Step 12: Feedback the task status to the master;

[0073] Step 13: Execute the task;

[0074] Step 14: Record the log information of the executed task in the database db;

[0075] Step 15: Feedback the task log information to the worker;

[0076] Step 16: The master sends a query request to the worker to request task execution progress information;

[0077] Step 17: After receiving the query request, the worker sends feedback to the master that the query request has been successfully received;

[0078] Step 18, the worker reports the task status to the master;

[0079] Step 19: The master sends feedback to the worker indicating that the task has been successfully received.

[0080] Step 20: Receive task status report;

[0081] Step 21: The Master receives the task instance status and log information (i.e., the task status report) and updates the task instance information in the database db.

[0082] Step 22: Database db gives feedback to master.

[0083] At the same time, the task push strategy can be expanded, including: 1) standlone mode (single-node mode), that is, assigned to a single worker for execution; 2) high-priority, second-level task instances, which can use the broadcast mode and notify all workers through Akka tell. When the worker receives the task instance, it will insert the task first according to the idle situation and execute the task in a rush; 3) the same task instance involves parallel nodes, and the map reduce mode can be adopted. Each worker executes separately, and finally merges and updates the task status to speed up the task execution.

[0084] It should be noted that the above steps 2 and 5 do not consider the time sequence, and querying task information and logs can be performed during the entire process of task execution.

[0085] The above embodiment proposes a distributed task scheduling method. The proposed distributed scheduling system based on Akka can efficiently realize the high-frequency and large-scale information transmission between the master and the worker, subverting the previous polling action; according to the health of the worker, it can reasonably allocate tasks and solve the previous polling action to obtain tasks; it supports three extended task modes, which can more efficiently solve the needs of diverse tasks, and the task execution in seconds is effectively guaranteed; because there is no polling action, the number of threads in the thread pool is greatly reduced; the entire distributed system is lock-free, which greatly reduces the loss, improves the system throughput, and can improve the problem of slow data output from the data warehouse.

[0086] Based on the distributed task push method provided in each of the above embodiments, based on the same inventive concept, a distributed task push device is also provided in this embodiment, which is used to implement the above embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0087] Figure 4 is a structural block diagram of a distributed task pushing device provided according to an embodiment of the present invention, such as Figure 4As shown, the device is applied to a distributed task push system, the system includes multiple master nodes and multiple task execution nodes, each master node manages at least one task execution node, and each task execution node executes one or more task instances of the task, and the above-mentioned device includes: an acquisition module 40, used to obtain one or more task instances of each of the multiple tasks to be pushed; a determination module 42, connected to the above-mentioned acquisition module 40, used to determine a task push strategy for distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node; wherein the task push strategy is formulated by the first quantity information of the multiple master nodes, the second quantity information of the multiple task execution nodes and the one or more task instances of each of the multiple tasks to be pushed; a distribution module 44, connected to the above-mentioned determination module 42, used to distribute the one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task push strategy.

[0088] Optionally, the determination module 42 includes: a broadcast unit, used to select any master node from multiple master nodes as the first master node, and broadcast the master node ID corresponding to the first master node to the remaining master nodes from the multiple master nodes except the first master node; a determination unit, used to screen out the second master node corresponding to the smallest or largest master node ID through any master node, and use the second master node as the leader node; a detection unit, used to detect the estimated execution time corresponding to executing multiple tasks through the leader node; a selection unit, used to select the first task push strategy in the task push strategy when the estimated execution time does not exceed a preset value; when the estimated execution time exceeds the preset value, select the second task push strategy in the task push strategy, wherein the second task push strategy is formulated by the first quantity information of multiple master nodes, the health factor corresponding to each master node, the second quantity information of multiple task execution nodes, one or more task instances of each of the multiple tasks to be pushed, and the health factor corresponding to each task execution node.

[0089] Optionally, the distribution module 44 includes: a first grouping unit, used to group multiple tasks according to a first grouping algorithm corresponding to a first task pushing strategy; a first pushing unit, used to push the multiple tasks to each master node in sequence according to the grouping result of the first grouping algorithm; a second pushing unit, used to select any one of the multiple master nodes as a third master node, and push one or more task instances corresponding to the tasks on the third master node to each task execution node managed by the third master node according to the first grouping algorithm; wherein the first grouping algorithm is to perform a modulo operation on the task ID and the first quantity information corresponding to each task, or to perform a modulo operation on the task instance ID and the second quantity information corresponding to each task instance.

[0090] Optionally, the distribution module 44 includes: a second grouping unit, used to group multiple tasks according to a second grouping algorithm corresponding to a second task push strategy; a third pushing unit, used to push the multiple tasks to each master node in sequence according to the grouping results corresponding to the second grouping algorithm; a fourth pushing unit, used to select any one of the multiple master nodes as the fourth master node, and push one or more task instances corresponding to the tasks on the fourth master node to each task execution node managed by the fourth master node according to the second grouping algorithm; wherein the second grouping algorithm is to perform a modulo operation on the task ID corresponding to each task and the health factor corresponding to each master node; or to perform a modulo operation on the task instance ID corresponding to each task instance and the health factor corresponding to each task execution node.

[0091] Optionally, the above-mentioned device also includes: a refresh module, which is used to refresh the health factors corresponding to each master node and each task execution node after the distribution module distributes one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task push strategy.

[0092] Optionally, the above-mentioned device also includes: a first receiving module, which is used to receive the log information and execution results corresponding to the execution of each task instance fed back by each task execution node after the distribution module distributes one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task pushing strategy, and after each task execution node executes each task instance.

[0093] Optionally, the above-mentioned device also includes: a sending module, which is used to send a query request to each master node to query the log information and execution progress information of each task execution node executing each task instance after the distribution module distributes one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task push strategy, and each master node sends the query request to each task execution node; a second receiving module, which is used to receive the log information and execution progress information fed back by each task execution node according to the query request.

[0094] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0095] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0096] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0097] S1, obtaining one or more task instances of each of the multiple tasks to be pushed;

[0098] S2, determining a task push strategy for distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node; wherein the task push strategy is formulated by the first quantity information of the multiple master nodes, the second quantity information of the multiple task execution nodes, and the one or more task instances of each of the multiple tasks to be pushed;

[0099] S3, distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task pushing strategy.

[0100] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0101] Based on the above Figure 1 The method shown and Figure 4 In order to achieve the above-mentioned purpose, the embodiment of the present invention also provides an electronic device, such as Figure 5 As shown, the system includes a memory 52 and a processor 51, wherein the memory 52 and the processor 51 are both arranged on a bus 53. The memory 52 stores a computer program, and the processor 51 executes the computer program to realize Figure 1 The distributed task push method shown.

[0102] Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a memory (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various implementation scenarios of the present invention.

[0103] Optionally, the device may also be connected to a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a Bluetooth interface, a WI-FI interface), etc.

[0104] Those skilled in the art will appreciate that the structure of an electronic device provided in this embodiment does not limit the physical device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0105] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0106] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0107] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A distributed task push method, characterized in that: Applied to a distributed task push system, the system includes multiple master nodes and multiple task execution nodes, each master node manages at least one task execution node, and each task execution node executes one or more task instances of a task, the method includes: Obtain one or more task instances of each of the multiple tasks to be pushed; Determine a task push strategy for distributing one or more task instances of each of the multiple tasks to be pushed to task execution nodes managed by each master node; wherein the task push strategy is formulated by the first quantity information of the multiple master nodes, the second quantity information of the multiple task execution nodes, and the one or more task instances of each of the multiple tasks to be pushed; Distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task pushing strategy; The task push strategy for determining to distribute one or more task instances of each of the plurality of tasks to be pushed to the task execution nodes managed by each master node includes: Selecting any one of the multiple master nodes as a first master node, and broadcasting a master node ID corresponding to the first master node to the remaining master nodes of the multiple master nodes except the first master node; Filter out the second master node corresponding to the smallest or largest master node ID through any master node, and use the second master node as the leader node; Detecting, by the leader node, the estimated execution time corresponding to executing the plurality of tasks; If the estimated execution time does not exceed the preset value, the first task push strategy among the task push strategies is selected; if the estimated execution time exceeds the preset value, the second task push strategy among the task push strategies is selected, wherein the second task push strategy is formulated by the first quantity information of the multiple master nodes, the health factor corresponding to each master node, the second quantity information of the multiple task execution nodes, one or more task instances of each of the multiple tasks to be pushed, and the health factor corresponding to each task execution node; The step of distributing one or more task instances of each of the plurality of tasks to be pushed to task execution nodes managed by each master node according to the task pushing strategy includes: grouping the multiple tasks according to a first grouping algorithm corresponding to the first task pushing strategy; Pushing the multiple tasks to each master node in sequence according to the grouping result corresponding to the first grouping algorithm; Selecting any one of the multiple master nodes as a third master node, and pushing one or more task instances of the task on the third master node to each task execution node managed by the third master node according to the first grouping algorithm; The first grouping algorithm is to perform a modulo operation on the task ID corresponding to each task and the first quantity information, or to perform a modulo operation on the task instance ID corresponding to each task instance and the second quantity information.

2. The method according to claim 1, characterized in that The step of distributing one or more task instances of each of the plurality of tasks to be pushed to task execution nodes managed by each master node according to the task pushing strategy includes: grouping the multiple tasks according to a second grouping algorithm corresponding to the second task pushing strategy; Pushing the multiple tasks to each master node in sequence according to the grouping result corresponding to the second grouping algorithm; Selecting any one of the multiple master nodes as a fourth master node, and pushing one or more task instances corresponding to the tasks on the fourth master node to each task execution node managed by the fourth master node according to the second grouping algorithm; Among them, the second grouping algorithm is to perform a modulo operation on the task ID corresponding to each task and the health factor corresponding to each master node; or to perform a modulo operation on the task instance ID corresponding to each task instance and the health factor corresponding to each task execution node.

3. The method according to claim 2, characterized in that After distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task pushing strategy, the method further includes: Refresh the health factors corresponding to each master node and each task execution node.

4. The method according to any one of claims 1 to 3, characterized in that After distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task pushing strategy, the method further includes: After each task execution node executes each task instance, log information and execution results corresponding to the execution of each task instance fed back by each task execution node are received.

5. The method according to any one of claims 1 to 3, characterized in that: After distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task pushing strategy, the method further includes: Sending a query request to each master node to query the log information and execution progress information of each task instance executed by each task execution node, and each master node sends the query request to each task execution node; The log information and the execution progress information fed back by each task execution node according to the query request are received.

6. A distributed task pushing device, characterized in that: Applied to a distributed task push system, the system includes multiple master nodes and multiple task execution nodes, each master node manages at least one task execution node, each task execution node executes one or more task instances of a task, and the device includes: An acquisition module, used to acquire one or more task instances of each of the multiple tasks to be pushed; A determination module, used to determine a task push strategy for distributing one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node; wherein the task push strategy is formulated by the first quantity information of the multiple master nodes, the second quantity information of the multiple task execution nodes, and the one or more task instances of each of the multiple tasks to be pushed; A distribution module, used to distribute one or more task instances of each of the multiple tasks to be pushed to the task execution nodes managed by each master node according to the task push strategy; The determination module includes: a broadcast unit, which is used to select any master node from a plurality of master nodes as a first master node, and broadcast the master node ID corresponding to the first master node to the remaining master nodes from the plurality of master nodes except the first master node; a determination unit, which is used to screen out the second master node corresponding to the smallest or largest master node ID through any master node, and use the second master node as a leader node; a detection unit, which is used to detect the estimated execution time corresponding to executing a plurality of tasks through the leader node; a selection unit, which is used to select the first task push strategy in the task push strategy when the estimated execution time does not exceed a preset value; and select the second task push strategy in the task push strategy when the estimated execution time exceeds the preset value, wherein the second task push strategy is formulated by the first quantity information of the plurality of master nodes, the health factor corresponding to each master node, the second quantity information of the plurality of task execution nodes, one or more task instances of the plurality of tasks to be pushed, and the health factor corresponding to each task execution node; The distribution module includes: a first grouping unit, used to group multiple tasks according to a first grouping algorithm corresponding to a first task pushing strategy; a first pushing unit, used to push the multiple tasks to each master node in sequence according to the grouping result of the first grouping algorithm; a second pushing unit, used to select any master node from the multiple master nodes as a third master node, and push one or more task instances corresponding to the tasks on the third master node to each task execution node managed by the third master node according to the first grouping algorithm; wherein the first grouping algorithm is to perform a modulo operation on the task ID and the first quantity information corresponding to each task, or to perform a modulo operation on the task instance ID and the second quantity information corresponding to each task instance.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images