Task processing method, device, equipment and storage medium

Abstract

The present application discloses a task processing method, device, equipment and storage medium, which are applied to a task processing cluster including multiple task processing nodes. The method is applied to the first task processing node in the task processing cluster, and belongs to the field of Internet technology. The method includes: obtaining a first task to be processed; determining the associated tasks of the first task based on the task chain; generating multiple first task slices in response to the completion of the associated task processing; and sending the first task slice to the target task processing node in the task processing cluster. The solution provided by the present application avoids the limitations of the timed processing of tasks to be processed in the related technology, making the processing conditions of the tasks to be processed more flexible; and the various task processing nodes in the task processing cluster are the same, and there is no need to configure additional nodes to send task slices to the task processing nodes. The task processing nodes send task slices to other task processing nodes, reducing the setting overhead of redundant nodes.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a task processing method, device, equipment and storage medium. Background Art

[0002] At present, business systems often include various tasks with large data processing and computing scales. In related technologies, for large-scale tasks, distributed task processing nodes can be used to process the large-scale tasks (such as order data statistics tasks). Summary of the invention

[0003] The embodiments of the present application provide a task processing method, apparatus, device and storage medium, which can make the processing conditions of the tasks to be processed more flexible. The technical solution is as follows:

[0004] According to one aspect of an embodiment of the present application, a task processing method is provided, which is applied to a task processing cluster including a plurality of task processing nodes. The method is applied to a first task processing node in the task processing cluster, and the method includes:

[0005] Get the first task to be processed;

[0006] Determine an associated task of the first task based on a task chain, where the task chain is used to describe a dependency relationship between different tasks, and the associated tasks include tasks on which the first task depends;

[0007] In response to completion of the associated task processing, generating a plurality of first task slices, where the first task slices are obtained by dividing the first task;

[0008] The first task slice is sent to a target task processing node in the task processing cluster.

[0009] According to one aspect of an embodiment of the present application, a task processing device is provided, which is applied to a task processing cluster including a plurality of task processing nodes. The device is applied to a first task processing node in the task processing cluster, and the device includes:

[0010] A task acquisition module, used to acquire a first task to be processed;

[0011] A task determination module, configured to determine associated tasks of the first task based on a task chain, wherein the task chain is used to describe a dependency relationship between different tasks, and the associated tasks include tasks on which the first task depends;

[0012] A slice generation module, configured to generate a plurality of first task slices in response to completion of the associated task processing, wherein the first task slices are obtained by dividing the first task;

[0013] The slice sending module is used to send the first task slice to the target task processing node in the task processing cluster.

[0014] According to one aspect of an embodiment of the present application, a computer device is provided, the computer device comprising a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-mentioned task processing method.

[0015] According to one aspect of an embodiment of the present application, a non-temporary computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned task processing method is implemented.

[0016] According to one aspect of an embodiment of the present application, a computer program product is provided. When the computer program product is executed, a computer device executes the above-mentioned task processing method.

[0017] The technical solution provided in the embodiments of the present application can bring the following beneficial effects:

[0018] The associated tasks of the first task are determined through the task chain, and after the associated tasks are processed, the first task to be processed is divided and processed, that is, the first task processing node can process the first task after the associated tasks of the first task are processed, so that the first task processing node can determine whether to process the first task according to the processing status of the associated tasks, thereby avoiding the limitations of the timed processing of tasks to be processed in the related technology, and making the processing condition setting of the tasks to be processed more flexible; and the task processing nodes in the task processing cluster are the same, and there is no need to configure additional nodes to send task slices to the task processing nodes. The task processing nodes send task slices to other task processing nodes, thereby reducing the setting overhead of redundant nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a schematic diagram of a task processing cluster provided by an embodiment of the present application;

[0021] Figure 2 is a flowchart of a task processing method provided by an embodiment of the present application;

[0022] Figure 3is a flowchart of a task processing method provided by another embodiment of the present application;

[0023] Figure 4 A schematic diagram showing an exemplary transmission method of task slicing and task processing results;

[0024] Figure 5 is a block diagram of a task processing device provided by an embodiment of the present application;

[0025] Figure 6 is a block diagram of a task processing device provided by another embodiment of the present application;

[0026] Figure 7 It is a structural block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0028] In the related art, when executing pending tasks, a management node is set up, and the management node is used to manage each node in the task processing cluster when obtaining pending tasks. For example, after obtaining the pending tasks, the management node processes the pending tasks regularly and sends the pending tasks to the corresponding nodes in the task processing cluster.

[0029] However, in the above-mentioned related technologies, the management node decides whether to process the pending task only based on time, and the setting method of the processing conditions of the pending task is single. If the pending task depends on other tasks, the processing conditions of the pending task can only be set by estimating the task processing time of the other tasks, and the processing conditions are not accurately set.

[0030] In the technical solution provided by the present application, the dependency relationship between each task is described by a task chain. After obtaining the first task, the task processing node in the task processing cluster determines the associated task that has a dependency relationship with the first task based on the task chain, and after the associated task is processed, determines to process the above-mentioned first task, thereby realizing the determination of whether the processing condition of the first task is met based on the associated task, and setting the processing condition to the completion of the associated task processing. The setting of the processing condition is flexible and accurate. In addition, each task processing node in the task processing cluster is the same, and can be used as a task initiation node to divide the first task, or as a task execution node to process the task slice, reducing the setting overhead of redundant management nodes.

[0031] Please refer to Figure 1, which shows a schematic diagram of a task processing cluster provided by an embodiment of the present application. The task processing cluster may include multiple task processing nodes 10.

[0032] The task processing node 10 refers to any node of the task processing system. Optionally, the task processing node 10 can be a computer device, a backend server, etc. In an embodiment of the present application, the task processing node 10 can be used to divide the tasks to be processed, process task slices, and send cluster information of the task cluster to other task processing nodes. Among them, the cluster information is used to indicate the task processing nodes included in the above-mentioned task processing cluster.

[0033] In a possible implementation, the above-mentioned task processing node 10 is a task initiating node. Optionally, after obtaining the task to be processed, the task processing node 10 detects whether the task to be processed meets the execution condition. If the task to be processed meets the execution condition, the task to be processed is divided to obtain a plurality of task slices corresponding to the task to be processed, and then the plurality of task slices are sent to the target task processing node. Among them, the target task processing node can be any task processing node in the task processing cluster, and the number of target task processing nodes can be one or more, which is not limited in the embodiment of the present application. Afterwards, after receiving the plurality of task processing results corresponding to the plurality of task slices sent by the target task processing node, the task processing node 10 aggregates and summarizes the plurality of task processing results to obtain the processing results corresponding to the above-mentioned task to be processed.

[0034] In another possible implementation, the task processing node 10 is a task execution node. Optionally, after receiving the task slice sent by the target task processing node, the task processing node 10 processes the task slice. If the task processing node 10 includes other task slices being processed, the task slice is processed based on the thread pool. Afterwards, after obtaining the task processing result corresponding to the task slice, the task processing node 10 sends the task processing result to the target task processing node.

[0035] In another possible implementation, the task processing node 10 is a seed node. Optionally, each task processing node in the task processing cluster stores a configuration file, which is used to indicate the seed node in the task processing cluster. In the case of a new node in the task processing cluster, the new node sends a cluster information acquisition request to the task processing node 10, and correspondingly, after obtaining the cluster information acquisition request, the task processing node 10 sends cluster information to the new node. In one possible implementation, the new node is a task processing node in the task processing cluster that is changed from a shutdown state to a non-shutdown state; in another possible implementation, the new node is a task processing node newly added when the task processing cluster is expanded. Optionally, after obtaining the above cluster information, the new node sends status information to each task processing node based on the cluster information, and the status information is used to indicate that the new node is in a non-shutdown state. Further, after obtaining the above status information, each task processing node can send task slices to the new node.

[0036] It should be noted that, in the embodiment of the present application, any task processing node in the task processing cluster can be interchanged between the three roles of the task initiation node, the task execution node and the seed node. For example, for any task processing node, when receiving a task to be processed, it is a task initiation node, when receiving a task slice, it is a task execution node, and when receiving a cluster information acquisition request, it is a seed node.

[0037] Optionally, the task processing nodes in the task processing cluster may communicate with each other via a network.

[0038] It must be noted that the above is introduced using the task processing cluster as a background server cluster as an example. In actual application, the various task processing nodes in the above task processing cluster can also be multiple virtual machines set on the same computer device, and the embodiments of the present application do not limit this.

[0039] The technical solution of the present application will be described in detail below in conjunction with several embodiments.

[0040] Please refer to Figure 2 , which shows a flowchart of a task processing method provided by an embodiment of the present application, which is applied to a task processing cluster including multiple task processing nodes, and the method is applied to the first task processing node in the task processing cluster. The execution subject of each step can be a computer device, such as Figure 1 The task processing node 10 in the task processing cluster (hereinafter referred to as the "first task processing node"). The method may include the following steps (201-205):

[0041] Step 201, obtain the first task to be processed.

[0042] The first task refers to any task received and not processed by the first task processing node. Optionally, the first task can be a data processing task such as data traversal and data conversion; or, the first task can also be a business processing task such as order processing and order storage; of course, in actual application, the above-mentioned first task can also be other tasks, which is not limited in the embodiment of the present application. In the embodiment of the present application, the first task processing node can receive the above-mentioned first task through the client, and the first task can include task data to be processed. Among them, the above-mentioned client can be a client used by ordinary users, or it can be a client used by staff corresponding to a certain job, which is not limited in the embodiment of the present application.

[0043] In a possible implementation, the first task processing node acquires the first task in real time. Optionally, upon receiving the first task, the first task processing node determines to acquire the first task to be processed, and further, the first task processing node monitors the processing conditions of the first task in real time, and processes the first task when the processing conditions of the first task are met.

[0044] In another possible implementation, the first task processing node obtains the first task when the processing condition corresponding to the first task is met. Optionally, after receiving the first task, the first task processing node stores the first task and monitors the processing condition corresponding to the first task. When the processing condition corresponding to the first task is met, the first task to be processed is obtained and the first task is processed.

[0045] The processing condition is a judgment condition for determining whether to process a task. Optionally, different tasks correspond to different task processing conditions. For example, the processing condition corresponding to the first task can be that the associated task processing is completed, and the processing condition corresponding to the second task is 8:00 am.

[0046] Step 202: Determine the associated tasks of the first task based on the task chain.

[0047] The task chain is used to describe the dependency relationship between different tasks. In one possible implementation, the tasks in the task chain may include tasks received by the first task processing node, and may also include tasks not received by the first task processing node, that is, the task chain is set in different task processing nodes in the same form. After receiving the above-mentioned first task, each task processing node can obtain the associated tasks of the first task based on the task chain, and the associated tasks can be processed and completed in any task processing node in the task processing cluster. In this case, when each task processing node completes a task, it can communicate with each task processing node and send the processing completion information of the task to each task processing node. In another possible implementation, the tasks in the task chain only include tasks received by the first task processing node, that is, the task chain is set in different forms in different task processing nodes. At this time, each task processing node has different task receiving permissions. After receiving the first task, the first task processing node obtains the associated tasks of the first task based on the task chain set by itself, and the associated tasks can only be processed and completed in the first task processing node.

[0048] Optionally, the above-mentioned task chain represents the dependency relationship between each task in the form of a chain. For example, if task B needs to apply the processing result of task A during the processing, it is determined that task B depends on task A, and task A is dependent on task B. At this time, task B is processed after task A, and task B is set after task A in the task chain. And so on, based on the dependency relationship between each task, each task is connected in the form of a chain to generate a task chain. It should be noted that in the embodiment of the present application, a task can depend on one or more other tasks, and correspondingly, a task can also be dependent on one or more other tasks.

[0049] One point that needs to be explained is that, in an embodiment of the present application, each task processing node of the above-mentioned task processing cluster includes a task chain. In one possible implementation, different task processing nodes include the same task chain, that is, each task processing node can receive any task, and each task processing node has the same task receiving authority. In another possible implementation, different task processing nodes include different task chains, that is, each task processing node has different task receiving authorities, different task processing nodes have different tasks, and a certain task processing node can only receive a certain task. Optionally, in an embodiment of the present application, the above-mentioned task chain can be set in the task processing node by the staff, and the staff can flexibly set the above-mentioned task chain based on work experience. Of course, the staff can also modify the set task chain according to the actual situation.

[0050] In an embodiment of the present application, after obtaining the above-mentioned first task to be processed, the first task processing node determines the associated tasks of the first task based on the above-mentioned task chain. The associated tasks include the tasks on which the first task depends. Optionally, the associated tasks may be tasks that need to be completed before the first task is processed, for example, the processing results of the associated tasks need to be used during the first task processing. It should be noted that the number of the above-mentioned associated tasks may be one or more, and the embodiment of the present application does not limit this.

[0051] Step 203: In response to completion of associated task processing, a plurality of first task slices are generated.

[0052] The first task slice is obtained by dividing the first task. In an embodiment of the present application, after obtaining the above-mentioned associated task, the first task processing node determines whether to start processing the above-mentioned first task based on the associated task. Optionally, after the above-mentioned associated task processing is completed, the first task processing node determines that the processing conditions corresponding to the first task are met, starts processing the above-mentioned first task, divides the first task, and generates multiple first task slices. Among them, the first task slice refers to a task slice obtained based on the division processing of the above-mentioned first task, including part of the content of the first task. Optionally, the content included in different first task slices may be different. For example, the above-mentioned first task includes both a processing task for data A and a processing task for data B. After the first task is divided, one first task slice may include a processing task for data A, and another first task slice may include a processing task for data B.

[0053] One point that needs to be explained is that, in the embodiment of the present application, if there are multiple associated tasks, when all the multiple associated tasks are completed, it is determined that the associated tasks are completed, and the processing of the first task begins.

[0054] Optionally, in an embodiment of the present application, the first task processing node may divide the first task based on a sharding rule. The sharding rule may be set by a staff member in each task processing node of the task processing cluster. Optionally, the above step 203 includes the following steps:

[0055] 1. Determine shard parameters based on sharding rules;

[0056] The sharding rule is used to indicate the way in which the first task processing node divides the first task. Optionally, different tasks may have the same sharding rule or different sharding rules, which is not limited in the embodiment of the present application.

[0057] In an embodiment of the present application, after determining that the above-mentioned associated task processing is completed, the first task processing node obtains the sharding rule corresponding to the first task, determines the sharding parameters based on the sharding rule, and determines the division method of the first task.

[0058] 2. Divide the first task according to the sharding parameters to generate multiple first task shards.

[0059] The slicing parameter is used to indicate the parameter corresponding to the first task slicing. The parameter may include a quantity parameter or a data volume parameter. Optionally, after determining the slicing parameter, the first task processing node divides the first task according to the slicing parameter to generate a plurality of first task slicings.

[0060] Among them, the above-mentioned shard parameters include the number of shards and / or the amount of shard data. The number of shards is used to indicate the upper limit value of the number of the above-mentioned first task shards, and the amount of shard data is used to indicate the upper limit value of the data amount corresponding to the task data to be processed included in the above-mentioned first task shard.

[0061] In a possible implementation, the sharding parameter includes the number of shards. Optionally, after determining that the associated task processing is completed, the first task processing node divides the first task based on the number of shards to obtain the plurality of first task shards. The number of the first task shards is less than or equal to the number of shards.

[0062] In another possible implementation, the slicing parameters include the slicing data volume. Optionally, after determining that the associated task processing is completed, the first task processing node divides the first task based on the slicing data volume to obtain the plurality of first task slicings. The data volume corresponding to the task processing to be processed included in the first task slicing is less than or equal to the slicing data volume.

[0063] Of course, in actual use, the above-mentioned sharding parameters may also include the number of shards and the amount of sharding data. For example, when the amount of data corresponding to the task data to be processed included in the above-mentioned first task is less than a threshold, the first task is divided based on the number of shards; when the amount of data corresponding to the task data to be processed included in the above-mentioned first task is greater than or equal to the threshold, the first task is divided based on the amount of sharding data.

[0064] Step 204: Send the first task slice to a target task processing node in the task processing cluster.

[0065] The target task processing node refers to the task processing node in the above-mentioned task processing cluster. The number of the target task processing nodes is less than or equal to the number of the above-mentioned first task slices. In an embodiment of the present application, after generating the first task slice, the first task processing node sends the first slice to the target task processing node. The above-mentioned first task slice carries the task data to be processed included in the first task slice when sending. Optionally, since the first task processing node can process the task slice, the above-mentioned target task processing node can include the first task processing node.

[0066] It should be noted that, in the embodiment of the present application, the first task processing node can determine the sending method of the first task slice based on the operation information of the target task processing node, wherein the operation information is used to indicate the current task processing capability of the target task processing node.

[0067] In a possible implementation, the first task processing node determines the slice sending frequency according to the above-mentioned operation information, and sends the first task slice to the above-mentioned target task processing node based on the slice sending frequency. Optionally, after generating the above-mentioned first task slice, the first task processing node obtains the operation information of the target task processing node, and determines the slice sending frequency based on the operation information of the target task processing node. Further, according to the slice sending frequency, the first task slice is sent to the target task processing node. Among them, the slice sending frequency is used to indicate the task slice sending rate of the first task processing node for the above-mentioned target task processing node. Optionally, different target task processing nodes have different corresponding slice sending rates. It should be noted that in an embodiment of the present application, the first task processing node can obtain the operation information of each task processing node in the task processing cluster in real time after generating the first task slice; or, the first task processing node can also communicate with each task processing node in the task processing cluster at a certain time interval to obtain the operation information of each task processing node.

[0068] In another possible implementation, the first task processing node determines the order of the target task processing node according to the above-mentioned operation information, and sends the first task slice to the target task processing node in sequence based on the order. Optionally, after generating the above-mentioned first task slice, the first task processing node obtains the operation information of the target task processing node, and determines the current task processing capacity of the target task processing node based on the operation information of the target task processing node. Further, according to the order of the current task processing capacity from high to low, the above-mentioned first task slice is sent to the target task processing node in sequence. Among them, the order of sending the first task slice is positively correlated with the current task processing capacity of the target task processing node. It should be noted that in the embodiment of the present application, the first task processing node can obtain the operation information of each task processing node in the task processing cluster in real time after generating the first task slice; or, the first task processing node can also communicate with each task processing node in the task processing cluster at a certain time interval to obtain the operation information of each task processing node.

[0069] It should be noted that, in the embodiment of the present application, after sending the first task slice, the first task processing node needs to wait for receiving the task processing result corresponding to the first task slice and aggregate the task processing result. Optionally, the above step 204 also includes the following sub-steps:

[0070] 1. Receive task processing results corresponding to multiple first task slices sent by the target task processing node.

[0071] The task processing result is used to indicate the processing result corresponding to the first task slice. For example, the task processing result is used to indicate whether the first task slice is successfully processed. Optionally, the task processing result carries the task data after the first task slice is processed.

[0072] In the embodiment of the present application, after sending the above-mentioned multiple first task slices to the target task processing node, the first task processing node receives the task processing results corresponding to the multiple first task slices sent by the target task processing node.

[0073] 2. Aggregate and summarize the processing results of multiple tasks to obtain the processing result corresponding to the first task.

[0074] Optionally, after obtaining the above-mentioned task processing results, the first task processing node aggregates and summarizes the multiple task processing results to obtain the processing result corresponding to the above-mentioned first task.

[0075] To sum up, in the technical solution provided by the embodiment of the present application, the associated tasks of the first task are determined through the task chain, and after the associated tasks are processed, the first task to be processed is divided and processed, that is, the first task processing node can process the first task after the associated tasks of the first task are processed, so that the first task processing node can determine whether to process the first task according to the processing status of the associated tasks, thereby avoiding the limitations of the timed processing of tasks to be processed in the related technology, and making the processing conditions of the tasks to be processed more flexible; and the task processing nodes in the task processing cluster are the same, and there is no need to configure additional nodes to send task slices to the task processing nodes. The task processing nodes send task slices to other task processing nodes, thereby reducing the setting overhead of redundant nodes.

[0076] In addition, the corresponding shard sending frequency is determined according to the operation information of the target task processing node, and different shard sending frequencies are determined for different target task processing nodes to avoid excessive load on the target task processing node caused by inappropriate shard sending rate.

[0077] In addition, according to the running information of the target task processing node, the order of sending task slices for different target task processing nodes is determined, and the task slices are sent first to the target task processing node with large current task processing capacity to ensure load balancing of each task processing node.

[0078] In addition, the task processing results corresponding to each first task slice are aggregated and summarized to obtain the processing result corresponding to the first task, which is convenient for subsequent query and application of the processing result of the first task.

[0079] In an exemplary embodiment, the first task processing node may include a task queue. The task queue is used to store unprocessed tasks received by the first task processing node. Optionally, the storage location of each task in the task queue corresponds to the task chain, and the first task processing node can determine whether to process the first task based on the task queue. For example, in the task queue, the associated task of the first task is stored before the first task. The first task processing node directly obtains the first task stored in the next position from the task queue after the associated task processing is completed based on the storage order of the tasks in the task queue. There is no need to judge whether the processing conditions of the first task are met, thereby improving the task processing efficiency and reducing the processing overhead of the first task processing node.

[0080] Optionally, in an embodiment of the present application, after obtaining the above-mentioned first task to be processed, the first task processing node determines the storage position of the first task in the task queue based on the above-mentioned task chain, and stores the first task in the task queue according to the storage position. The storage position of the first task is located after the above-mentioned associated task. Further, after the first task processing node obtains the associated task of the above-mentioned first task, the processing result of the associated task is monitored, and when the processing result of the associated task is detected, it is determined that the associated task processing is completed. At this time, if the above-mentioned first task is at the first place in the task queue, the first task is divided and processed.

[0081] It should be noted that the above is introduced from the perspective of the first task processing node as the task initiating node. In actual application, the first task processing node can also be used as a task execution node to receive task slices sent by other task processing nodes for processing. The specific steps are as follows:

[0082] 1. Receive a second task slice from a second task processing node.

[0083] The second task processing node refers to any task processing node in the task processing cluster. Optionally, the second task processing node may be another task processing node in the task processing cluster except the first task processing node.

[0084] The second task slice is obtained by dividing the second task. Optionally, after obtaining the second task, the second task processing node obtains the associated task of the second task based on the task chain, and after the associated task is processed, the second task is divided to obtain the above-mentioned second task slice, and then the second task slice is sent to the first task processing node. The second task slice may carry the task data to be processed.

[0085] Correspondingly, the first task processing node receives the second task slice from the second task processing node.

[0086] 2. If the first task processing node is processing other task slices, the second task slice is processed in parallel based on the thread pool.

[0087] The thread pool is used to implement parallel processing of multiple task slices. In the embodiment of the present application, the thread pool is used to implement parallel processing of multiple task slices. During the execution of the task slices, the existing threads are used to reduce the overhead of thread creation and extinction, and the performance is good. Moreover, the thread pool can effectively control the maximum number of concurrent threads, improve the resource utilization rate of the first task processing node, avoid excessive resource competition, and avoid congestion.

[0088] Optionally, after receiving the second task slice, the first task processing node monitors its own task processing status. If the first task processing node is processing other task slices, the second task slice is processed in parallel based on the thread pool.

[0089] 3. In response to completion of processing of the second task slice, a task processing result corresponding to the second task slice is sent to the second task processing node.

[0090] In the following embodiment of the present application, after the first task processing node completes processing of the second task slice, a task processing result corresponding to the second task slice is generated, and the task processing result corresponding to the second task slice is sent to the second task processing node. The task processing result carries the task data after the second task slice is processed.

[0091] Exemplary, in conjunction with reference Figure 3 , a complete introduction to the task processing method provided in this application is given.

[0092] Step 301: A first task processing node obtains a first task to be processed.

[0093] Optionally, the first task processing node obtains the first task to be processed through a client, wherein the client may be a client used by an ordinary user or a client used by a staff member.

[0094] Step 302: The first task processing node stores the first task in a task queue based on the task chain.

[0095] The task chain is used to describe the dependency between different tasks, that is, the task chain can be used to describe the dependency between the first task and other tasks. In an embodiment of the present application, after acquiring the above-mentioned first task, the first task processing node determines the storage location of the first task in the task queue based on the task chain, and stores the first task in the task queue. Wherein, the storage location of the first task is located after the associated task.

[0096] Step 303: when the first task is at the first position in the task queue, the first task is used as a task to be processed, and the first task processing node obtains the task to be processed from the first position in the task queue.

[0097] In an embodiment of the present application, the first task processing node sequentially obtains pending tasks from the task queue for processing according to the task queue, and obtains the next pending task for processing after the previous pending task is processed. Optionally, when the first task is at the top of the task queue, the first task processing node takes the first task as the pending task, and the first task processing node obtains the pending task from the top of the task queue.

[0098] Step 304, the first task processing node determines whether the processing condition of the task to be processed is met. If so, step 305 is executed; if not, step 304 is executed.

[0099] The processing condition refers to the judgment condition for determining whether to process the above-mentioned pending tasks. Optionally, different pending tasks correspond to different processing conditions. For example, the processing condition of the first task may be that all the associated tasks of the first task are processed, and the processing condition of another task may be that the processing time is 8 am.

[0100] In the embodiment of the present application, after obtaining the above-mentioned task to be processed, the first task processing node determines the processing condition of the task to be processed. If the processing condition is met, the task to be processed begins to be processed; otherwise, the processing condition continues to be monitored until the processing condition is met.

[0101] Step 305: The first task processing node divides the task to be processed and generates task slices corresponding to the task to be processed.

[0102] In the embodiment of the present application, after determining that the above processing conditions are met, the first task processing node divides the task to be processed and generates task slices corresponding to the task to be processed.

[0103] Step 306: The first task processing node sends the task slice corresponding to the task to be processed to the target task processing node.

[0104] Correspondingly, the target task processing node receives the task slice corresponding to the task to be processed.

[0105] Step 307: The target task processing node processes the task slices corresponding to the task to be processed based on the thread pool.

[0106] Optionally, after receiving the task slices corresponding to the above-mentioned pending task, the target task processing node processes the task slices corresponding to the pending task based on the thread pool. Wherein, the thread pool is used to realize the parallel processing of multiple task slices. In other words, if the target task processing node is processing other task slices, then based on the thread pool, the task slices corresponding to the pending task can be processed simultaneously, avoiding the processing overhead caused by task waiting.

[0107] Step 308: After the task slice corresponding to the task to be processed is processed, the target task processing node sends the task processing result of the task slice corresponding to the task to be processed to the first task processing node.

[0108] Correspondingly, the first task processing node receives the task processing result.

[0109] Step 309: The first task processing node aggregates and summarizes the processing results of the task slices corresponding to the task to be processed.

[0110] Step 310: The first task processing result determines whether the aggregation and summary of the task slices corresponding to the task to be processed is completed. If yes, execute step 311; if no, execute step 309.

[0111] Optionally, when aggregation and summary of task slices corresponding to the task to be processed are completed, the first task processing node determines that processing of the task to be processed is completed.

[0112] Step 311, the first task processing node determines whether the task queue is empty. If so, the process ends; if not, step 312 is executed.

[0113] Optionally, in an embodiment of the present application, after determining that the to-be-processed task is completed, the first task processing node determines the task queue and if no other tasks exist in the task queue, determines that the task queue is empty and ends the process.

[0114] Step 312: The first task processing node obtains other tasks from the first position of the task queue, and uses the other tasks as tasks to be processed, and executes the above steps 304-311 in a loop.

[0115] The other tasks mentioned above may be dependent on the first task mentioned above, that is, the first task may be an associated task of the other tasks.

[0116] Of course, in the embodiment of the present application, since the first task slice carries the task data to be processed, and the task processing result of the first task slice carries the task data that has been processed, the first task slice and the task processing result of the first task slice need to be serialized and converted into a transmittable and storable mode before transmission.

[0117] Exemplary, in conjunction with reference Figure 4After obtaining the first task slice 40, the first task processing node serializes the first task slice 40 through the conversion layer to obtain the serialized first task slice 41, and stores the serialized first task slice 41 in the buffer layer, and then transmits the serialized first task slice 41 through the transport layer. Correspondingly, after the target task processing node receives the serialized first task slice 41 through the transport layer, the buffer layer stores the serialized first task slice 41, and deserializes the serialized first task slice 41 through the conversion layer to obtain the first task slice 40. Further, the target task processing node processes the first task slice 40 to obtain the task processing result 42 corresponding to the first task slice. Similarly, the target task processing node serializes the task processing result 42 through the conversion layer to obtain the serialized task processing result 43, and stores the serialized task processing result 43 in the buffer layer, and then transmits the serialized task processing result 43 through the transport layer. Correspondingly, after the first task processing node receives the serialized task processing result 43 through the transport layer, the buffer layer stores the serialized task processing result 43, and deserializes the serialized task processing result 43 through the conversion layer to obtain the task processing result 42. In addition, after the first task processing node completes the aggregation of the first task and obtains the processing result corresponding to the first task, the first task processing node and the target task processing node clear the storage of the first task in the buffer layer, such as clearing the first task slice and the task processing result corresponding to the first task slice.

[0118] It should be noted that in the embodiment of the present application, due to the influence of various environmental factors, the first task processing node may undergo a state transition. At this time, it is necessary to communicate with each node in the task processing cluster based on the third task processing node (seed node). The specific steps are as follows:

[0119] 1. In response to the first task processing node being converted from a shutdown state to a non-shutdown state, a cluster information acquisition request is sent to a third task processing node in the task processing cluster.

[0120] The shutdown state is used to indicate that the first task processing node is in an inoperable state. The non-shutdown state is used to indicate that the first task processing node is in an operable state. In an embodiment of the present application, when the first task processing node is converted from the shutdown state to the non-shutdown state, the first task processing node may send a cluster information acquisition request to the third task processing node, and the cluster information acquisition request is used to request the third task processing node to obtain cluster information.

[0121] Optionally, in an embodiment of the present application, each task processing node in the task processing cluster includes a configuration file, and the configuration file is used to indicate a third task processing node in the task processing cluster. In a possible implementation, when the first task processing node is converted from a shutdown state to a non-shutdown state, the first task processing node can determine the third task processing node based on the configuration file and send a cluster information acquisition request to the third task processing node.

[0122] Correspondingly, after receiving the cluster information acquisition request, the third task processing node sends cluster information to the first task processing node based on the cluster information acquisition request.

[0123] 2. When receiving the cluster information sent by the third task processing node, send the first status information to each task processing node according to the cluster information.

[0124] The cluster information is used to indicate the task processing nodes included in the task processing cluster. Optionally, the cluster information may include a unique identifier of each task processing node included in the task processing cluster, and the first task processing node may determine each task processing node included in the task processing cluster according to the unique identifier.

[0125] In an embodiment of the present application, after receiving the cluster information sent by the third task processing node, the first task processing node determines each task processing node in the task processing cluster according to the cluster information, and sends first status information to each task processing node. The first status information is used to indicate that the first task processing node is in a non-shutdown state.

[0126] One point that needs to be explained is that in the embodiment of the present application, after receiving the first status information from the first task processing node, each of the above-mentioned task processing nodes needs to send feedback information regarding the first status information to the first task processing node, and then, the first task processing node determines that each task processing node is in a non-shutdown state based on the feedback information.

[0127] 3. If no feedback information is received from other task processing nodes regarding the first status information, it is determined that the other task processing nodes are in a shutdown state.

[0128] In an embodiment of the present application, after the first task processing node sends the first status information to the above-mentioned task processing nodes, if no feedback information is received from other task processing nodes regarding the first status information, it is determined that the other task processing nodes are in a shutdown state. The above-mentioned other task processing nodes refer to any task processing node in the task processing cluster except the first task processing node.

[0129] In a possible implementation, each task processing node in the task processing cluster communicates at a certain time interval to obtain status information and operation information of each task processing node. Since each task processing node in the task processing cluster can determine that the other task processing nodes are in a shutdown state during the communication process, in order to save signaling transmission overhead, the first task processing node does not send second status information to each task processing node when determining that the other task processing nodes are in a shutdown state. The second status information is used to indicate that the other task processing nodes are converted to a shutdown state.

[0130] In another possible implementation, each task processing node in the task processing cluster communicates with each other task processing node only when it is converted to a non-shutdown state. Therefore, in order to ensure the accuracy of information obtained by each task processing cluster, the first task processing node sends the second state information to each task processing node when determining that other task processing nodes are in a shut down state. Correspondingly, after receiving the second state information, each task processing node stops sending task slices to the other task processing nodes.

[0131] Of course, in the embodiment of the present application, the above method can also be applied to the expansion process of the task processing cluster. For example, if the first task processing node is a new node newly put into use in the task processing cluster, the first task processing node can obtain cluster information based on the third task processing node, and send the first status information to each task processing node in the task processing cluster based on the cluster information.

[0132] One point that needs to be explained is that in the embodiment of the present application, any task processing node in the above-mentioned task processing cluster can be configured as a third task processing node, such as the first task processing node can also be configured as a third task processing node, and the embodiment of the present application does not limit this.

[0133] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0134] Please refer to Figure 5 , which shows a block diagram of a task processing device provided by an embodiment of the present application, which is applied to a task processing cluster including multiple task processing nodes, and the device is applied to the first task processing node in the task processing cluster. The device has the function of implementing the above-mentioned task processing method, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be a computer device, and can also be set in a computer device. The device 500 may include: a task acquisition module 501, a task determination module 502, a slice generation module 503 and a slice sending module 504.

[0135] The task acquisition module 501 is used to acquire a first task to be processed.

[0136] The task determination module 502 is used to determine the associated tasks of the first task based on a task chain, wherein the task chain is used to describe the dependency relationship between different tasks, and the associated tasks include the tasks on which the first task depends.

[0137] The slice generation module 503 is used to generate a plurality of first task slices in response to completion of the associated task processing, where the first task slices are obtained by dividing the first task.

[0138] The slice sending module 504 is used to send the first task slice to the target task processing node in the task processing cluster.

[0139] In an exemplary embodiment, the slice sending module 504 is used to determine the slice sending frequency based on the operating information of the target task processing node, and the slice sending frequency is used to indicate the task slice sending rate of the first task processing node to the target task processing node; according to the slice sending frequency, the first task slice is sent to the target task processing node; wherein the operating information is used to indicate the current task processing capability of the target task processing node.

[0140] In an exemplary embodiment, the slice sending module 504 is used to determine the current task processing capability of the target task processing node based on the operation information of the target task processing node; and send the first task slices to the target task processing node in order from high to low according to the current task processing capabilities.

[0141] In an exemplary embodiment, the shard generation module 503 is used to determine shard parameters based on shard rules; divide the first task according to the shard parameters to generate the multiple first task shards; wherein the shard parameters include the number of shards and / or the amount of shard data, and the shard data amount is used to indicate the upper limit value of the data amount corresponding to the task data to be processed included in the first task shard.

[0142] In an exemplary embodiment, if Figure 6 As shown, the device 500 further includes: a position determination module 505 , a task storage module 506 and a result determination module 507 .

[0143] The position determination module 505 is used to determine the storage position of the first task in the task queue based on the task chain. The task queue is used to store unprocessed tasks received by the first task processing node. The storage position of the first task is located after the associated task.

[0144] The task storage module 506 is used to store the first task in the task queue according to the storage location.

[0145] The result determination module 507 is used to determine that the processing of the associated task is completed when the processing result of the associated task is detected.

[0146] The slice generation module 503 is further configured to execute the step of generating a plurality of first task slices in response to the first task being at the top of the task queue.

[0147] In an exemplary embodiment, if Figure 6 As shown, the device 500 further includes: a result receiving module 508 and a result aggregation module 509 .

[0148] The result receiving module 508 is used to receive the task processing results corresponding to the plurality of first task slices sent by the target task processing node.

[0149] The result aggregation module 509 is used to aggregate and summarize the multiple task processing results to obtain the processing result corresponding to the first task.

[0150] In an exemplary embodiment, if Figure 5 As shown, the device 500 also includes: a task processing module 510.

[0151] The task processing module 510 is used to receive a second task slice from a second task processing node, where the second task slice is obtained by dividing the second task, and the second task slice carries task data to be processed; if the first task processing node is processing other task slices, the second task slice is processed in parallel based on a thread pool; in response to completion of processing of the second task slice, a task processing result corresponding to the second task slice is sent to the second task processing node, where the task processing result carries the task data after processing by the second task slice.

[0152] In an exemplary embodiment, if Figure 5 As shown, the device 500 further includes: a request sending module 511 and an information sending module 512.

[0153] The request sending module 511 is configured to send a cluster information acquisition request to a third task processing node in the task processing cluster in response to the first task processing node being converted from a shutdown state to a non-shutdown state.

[0154] The information sending module 512 is used to send first status information to each task processing node according to the cluster information when receiving the cluster information sent by the third task processing node, wherein the first status information is used to indicate that the first task processing node is in the non-shutdown state, and the cluster information is used to indicate the task processing nodes included in the task processing cluster.

[0155] In an exemplary embodiment, the information sending module 512 is also used to determine that the other task processing nodes are in the shutdown state if no feedback information is received from other task processing nodes regarding the first state information, and the other task processing nodes refer to any task processing nodes in the task processing cluster except the first task processing node; and send second state information to each task processing node, and the second state information is used to indicate that the other task processing nodes are converted to the shutdown state.

[0156] To sum up, in the technical solution provided by the embodiment of the present application, the associated tasks of the first task are determined through the task chain, and after the associated tasks are processed, the first task to be processed is divided and processed, that is, the first task processing node can process the first task after the associated tasks of the first task are processed, so that the first task processing node can determine whether to process the first task according to the processing status of the associated tasks, thereby avoiding the limitations of the timed processing of tasks to be processed in the related technology, and making the processing conditions of the tasks to be processed more flexible; and the task processing nodes in the task processing cluster are the same, and there is no need to configure additional nodes to send task slices to the task processing nodes. The task processing nodes send task slices to other task processing nodes, thereby reducing the setting overhead of redundant nodes.

[0157] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0158] Please refer to Figure 7 , which shows a structural block diagram of a computer device provided by an embodiment of the present application. The computer device can be used to implement the above-mentioned task processing method. Specifically:

[0159] The computer device 700 includes a central processing unit (CPU) 701, a system memory 704 including a random access memory (RAM) 702 and a read only memory (ROM) 703, and a system bus 705 connecting the system memory 704 and the central processing unit 701. The computer device 700 also includes a basic input / output (I / O) system 706 for facilitating information transmission between various components in the computer, and a mass storage device 707 for storing an operating system 713, application programs 714, and other program modules 715.

[0160] The basic input / output system 706 includes a display 707 for displaying information and an input device 709 such as a mouse and a keyboard for user inputting information. The display 708 and the input device 709 are connected to the central processing unit 701 through an input / output controller 710 connected to the system bus 705. The basic input / output system 706 may also include an input / output controller 710 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 710 also provides output to a display screen, a printer, or other types of output devices.

[0161] The mass storage device 707 is connected to the central processing unit 701 through a mass storage controller (not shown) connected to the system bus 705. The mass storage device 707 and its associated computer readable medium provide non-volatile storage for the computer device 700. That is, the mass storage device 707 may include a computer readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0162] Without loss of generality, computer readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 704 and mass storage device 707 can be collectively referred to as memory.

[0163] According to various embodiments of the present application, the computer device 700 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 700 can be connected to the network 712 through the network interface unit 711 connected to the system bus 705, or the network interface unit 711 can be used to connect to other types of networks or remote computer systems (not shown).

[0164] The memory also includes a computer program, which is stored in the memory and configured to be executed by one or more processors to implement the above-mentioned task processing method.

[0165] In an exemplary embodiment, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned task processing method is implemented.

[0166] Optionally, the computer readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives) or optical disks, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0167] In an exemplary embodiment, a computer program product is also provided. When the computer program product is executed, a computer device executes the above-mentioned task processing method.

[0168] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0169] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A task processing method, characterized in that: Applied to a task processing cluster including a plurality of task processing nodes, the method is applied to a first task processing node in the task processing cluster, and the method comprises: Get the first task to be processed; Determining a storage location of the first task in a task queue based on a task chain, the task queue being used to store unprocessed tasks received by the first task processing node, the storage location of the first task being located after an associated task of the first task, the task chain being used to describe a dependency relationship between different tasks, the associated tasks including tasks on which the first task depends; storing the first task in the task queue according to the storage location; In the case where the first task is at the first position in the task queue, taking the first task as a task to be processed, and obtaining the task to be processed from the first position in the task queue; In the case where the processing conditions of the task to be processed are met, the task to be processed is divided and processed to generate task slices corresponding to the task to be processed, and the processing conditions of the task to be processed include that the processing of the associated tasks of the task to be processed is completed; Sending the task slice corresponding to the task to be processed to the target task processing node in the task processing cluster; Receive the task processing result of the task slice corresponding to the task to be processed sent by the target task processing node; the target task processing node is used to process the task slice corresponding to the task to be processed based on the thread pool, and after the task slice corresponding to the task to be processed is processed, send the task processing result of the task slice corresponding to the task to be processed to the first task processing node, and the thread pool is used to realize parallel processing of multiple task slices; Aggregate and summarize the task processing results of the task slices corresponding to the to-be-processed tasks, and when the aggregation and summary of the task slices corresponding to the to-be-processed tasks is completed, determine whether the task queue is empty; When the task queue is empty, the task processing flow is terminated; When the task queue is not empty, other tasks are obtained from the first position of the task queue, and the other tasks are used as tasks to be processed, and the step of obtaining the tasks to be processed from the first position of the task queue is re-executed.

2. The method according to claim 1, characterized in that: The sending the task slice corresponding to the to-be-processed task to the target task processing node in the task processing cluster includes: Determine a slice sending frequency based on the operation information of the target task processing node, where the slice sending frequency is used to indicate a task slice sending rate of the first task processing node to the target task processing node; Sending the task slices corresponding to the task to be processed to the target task processing node according to the slice sending frequency; The operation information is used to indicate the current task processing capability of the target task processing node.

3. The method according to claim 1, characterized in that The sending the task slice corresponding to the to-be-processed task to the target task processing node in the task processing cluster includes: Determining the current task processing capability of the target task processing node based on the operation information of the target task processing node; According to the order of the current task processing capabilities from high to low, the task slices corresponding to the to-be-processed tasks are sent to the target task processing nodes in sequence.

4. The method according to claim 1, characterized in that: The generating of task slices corresponding to the task to be processed includes: Determine sharding parameters based on sharding rules; Divide the task to be processed according to the slicing parameters to generate task slices corresponding to the task to be processed; The slice parameters include the number of slices and / or the amount of slice data, and the amount of slice data is used to indicate an upper limit value of the amount of data corresponding to the task data to be processed included in the task slice.

5. The method according to claim 1, characterized in that: The method further comprises: receiving a second task slice from a second task processing node, where the second task slice is obtained by dividing the second task, and the second task slice carries task data to be processed; If the first task processing node is processing other task slices, the second task slices are processed in parallel based on the thread pool; In response to completion of processing of the second task slice, a task processing result corresponding to the second task slice is sent to the second task processing node, wherein the task processing result carries task data processed by the second task slice.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: In response to the first task processing node being converted from a shutdown state to a non-shutdown state, sending a cluster information acquisition request to a third task processing node in the task processing cluster; When the cluster information sent by the third task processing node is received, first status information is sent to each task processing node according to the cluster information, wherein the first status information is used to indicate that the first task processing node is in the non-shutdown state, and the cluster information is used to indicate the task processing nodes included in the task processing cluster.

7. The method according to claim 6, characterized in that After sending the first status information to each task processing node according to the cluster information, the method further includes: If no feedback information is received from other task processing nodes regarding the first state information, it is determined that the other task processing nodes are in the shutdown state, and the other task processing nodes refer to any task processing node in the task processing cluster except the first task processing node; Second status information is sent to each of the task processing nodes, where the second status information is used to instruct the other task processing nodes to switch to the shutdown state.

8. A task processing device, characterized in that: Applied to a task processing cluster including a plurality of task processing nodes, the device is applied to a first task processing node in the task processing cluster, and the device comprises: A task acquisition module, used to acquire a first task to be processed; a position determination module, configured to determine a storage position of the first task in a task queue based on a task chain, wherein the task queue is configured to store unprocessed tasks received by the first task processing node, the storage position of the first task is located after an associated task of the first task, the task chain is configured to describe a dependency relationship between different tasks, and the associated tasks include tasks on which the first task depends; A task storage module, used for storing the first task in the task queue according to the storage location; A slice generation module is used to, when the first task is at the first position of the task queue, take the first task as a task to be processed and obtain the task to be processed from the first position of the task queue; when the processing conditions of the task to be processed are met, divide the task to be processed and generate task slices corresponding to the task to be processed, wherein the processing conditions of the task to be processed include that the processing of the associated tasks of the task to be processed is completed; A slice sending module, used to send the task slice corresponding to the to-be-processed task to the target task processing node in the task processing cluster; A result receiving module, used for receiving the task processing result of the task slice corresponding to the task to be processed sent by the target task processing node; the target task processing node is used to process the task slice corresponding to the task to be processed based on the thread pool, and after the task slice corresponding to the task to be processed is processed, send the task processing result of the task slice corresponding to the task to be processed to the first task processing node, and the thread pool is used to realize parallel processing of multiple task slices; A result aggregation module is used to aggregate and summarize the task processing results of the task slices corresponding to the task to be processed. When the aggregation and summary of the task slices corresponding to the task to be processed is completed, it is determined whether the task queue is empty; when the task queue is empty, the task processing flow is terminated; when the task queue is not empty, other tasks are obtained from the first position of the task queue, and the other tasks are used as the task to be processed, and the step of obtaining the task to be processed from the first position of the task queue is re-executed.

9. A computer device, characterized in that: The computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images