A task processing method, device and equipment
Through task execution node cluster acquisition and processing of pending tasks in resource clusters, and using event listening and heartbeat maintenance mechanisms, the problems of single point failure and poor fault tolerance in traditional task system management are solved, and task processing efficiency and system self-healing are improved Capability, achieving high reliability and low maintenance costs.
Patent Information
- Application Number
- CN202311585099.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-24
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-11-24
AI Technical Summary
Traditional task system management has problems such as single point of failure, poor fault tolerance, lack of self-healing ability and high maintenance costs, especially in high reliability continuous systems.
Through the task execution node cluster, obtain target work declaration events in the resource cluster, determine pending tasks, and execute tasks according to the event type, use event listening mechanism and heartbeat maintenance mechanism to achieve independent distribution and collision avoidance of tasks, cancel traditional central nodes, and enhance Self-coordination ability in task processing.
It improves the efficiency of task processing and the system's self-healing ability, avoids single point of failure, reduces preemptive collisions of tasks to be processed, realizes automatic drift and self-healing of tasks, and reduces maintenance costs.
Smart Images

Figure CN118034965B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of communication technology, and in particular to a task processing method, device and equipment. Background Art
[0002] The traditional mode of task system management is mostly based on the central management node, which distributes and schedules tasks uniformly. When the central node fails, the task cluster is leaderless and cannot sustain itself. Task dispatch operations are concentrated on the management node. When the number of tasks is large, the management node is under great pressure and the task dispatch efficiency is average. There are defects such as single point failure, poor fault tolerance, lack of self-healing ability, high maintenance cost, and low automation ability. For high-reliability continuous systems, there are potential performance hazards and security risks. Summary of the invention
[0003] The technical problem to be solved by the present invention is to provide a task processing method, device and equipment to enhance the self-coordination ability during task processing and improve the task processing efficiency.
[0004] In order to solve the above technical problems, an embodiment of the present invention provides a task processing method, the method comprising:
[0005] Acquire at least one target work declaration event in a resource cluster through at least one task execution node in a task execution node cluster, wherein the resource cluster includes a plurality of task declaration nodes, and at least one target work declaration event is published in the resource cluster in a first format;
[0006] Determine at least one target pending task among a plurality of pending tasks of the target work declaration event;
[0007] At least one of the target tasks to be processed is executed according to the type of the target work declaration event.
[0008] Optionally, before obtaining at least one target work declaration event in the resource cluster, the method further includes:
[0009] Acquire a work declaration event list in the resource cluster in real time, and cache the work declaration event list locally;
[0010] A task list in the resource cluster is obtained in real time, and the task list is cached locally, wherein the task list includes multiple published unexecuted tasks and executed tasks, the published unexecuted tasks correspond to first attribute information, the executed tasks correspond to second attribute information, and the executed tasks correspond to identification information of the task execution node that executes the executed tasks.
[0011] Optionally, determining at least one target to-be-processed task among a plurality of to-be-processed tasks of the target work declaration event includes:
[0012] Determine the to-be-processed task interval of the target work declaration event according to the target attribute information of the target work declaration event and the historical attribute information in the resource cluster, wherein the target attribute information includes: the required number of to-be-processed tasks corresponding to the current target work declaration event, the first number information and the first name information corresponding to each required to-be-processed task, and the historical attribute information includes: the number of processed tasks corresponding to the historical work declaration event, the second number information of each processed task and the historical name information corresponding to the historical work declaration event;
[0013] The required tasks to be processed in the task interval to be processed are sorted to obtain at least one target task to be processed, each target task to be processed having a first random number information.
[0014] Optionally, when the type of the target work declaration event is to create a task, executing at least one of the target tasks to be processed according to the type of the target work declaration event includes:
[0015] According to the first name information and the first random number information of at least one of the target tasks to be processed, traverse the task list of the current task execution node, and when the first name information and the first random number information of any target task to be processed do not match the name information and the number information in the first attribute information in the task list, and do not match the name information and the number information in the second attribute information in the task list, generate first registration information, wherein the first registration information carries the first name information and the first random number information of the current target task to be processed and the identification information of the current task execution node;
[0016] According to the first registration information, a task execution number thread corresponding to the current target task to be processed is created locally.
[0017] Optionally, when the type of the target work declaration event is a deletion task, executing the target to-be-processed task in the target to-be-processed task interval according to the type of the target work declaration event includes:
[0018] According to the first name information and the first random number information of at least one of the target tasks to be processed, traverse the task list of the current task execution node, and when the first name information and the first random number information of any target task to be processed match the name information and the number information in the first attribute information in the task list, or match the name information and the number information in the second attribute information in the task list, generate deletion information, and the deletion information carries the first name information and the first random number information of the current target task to be processed;
[0019] According to the deletion information, the task execution number thread corresponding to the current target task to be processed is deleted locally.
[0020] Optionally, when the type of the target work declaration event is to delete a task, the method further includes:
[0021] When the identification information of the current task execution node matches the preset task execution node identification information of the current target task to be processed, the task execution number thread corresponding to the current target task to be processed is locally deleted;
[0022] When the identification information of the current task execution node does not match the preset task execution node identification information of the current target task to be processed, traverse the task list of the current task execution node according to the first name information and the first random number information of the current target task to be processed, and when the first name information and the first random number information of any target task to be processed do not match the name information and the number information in the first attribute information in the task list, and do not match the name information and the number information in the second attribute information in the task list, generate second registration information according to the first name information and the first random number information of the current target task to be processed and the identification information of the current task execution node;
[0023] The second registration information is sent to the resource cluster.
[0024] Optionally, after executing at least one of the target tasks to be processed, the method further includes:
[0025] The identification information of the task execution node executing the current target task to be processed, the first name information and the first random number information of the current target task to be processed are saved in the local task list, and sent to the resource cluster at the same time.
[0026] A task processing device, comprising:
[0027] An acquisition module, configured to acquire at least one target work declaration event in a resource cluster through at least one task execution node in a task execution node cluster, wherein the resource cluster includes a plurality of task declaration nodes, and at least one target work declaration event is published in the resource cluster in a first format;
[0028] The processing module is used to determine at least one target to-be-processed task among multiple to-be-processed tasks of the target work declaration event; and execute at least one of the target to-be-processed tasks according to the type of the target work declaration event.
[0029] An embodiment of the present invention also provides a communication device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in any one of the above items.
[0030] An embodiment of the present invention further provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute any of the above methods.
[0031] The above solution of the present invention includes at least the following beneficial effects:
[0032] The above-mentioned scheme of the present invention obtains at least one target work declaration event in a resource cluster through at least one task execution node in the task execution node cluster, the resource cluster includes multiple task declaration nodes, and at least one of the target work declaration events is published in the resource cluster in a first format; determines at least one target to-be-processed task among multiple to-be-processed tasks of the target work declaration event; and executes at least one of the target to-be-processed tasks according to the type of the target work declaration event, thereby enhancing the self-coordination capability during task processing and improving the task processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 A flowchart of a method for processing a task provided by an embodiment of the present invention;
[0034] Figure 2 A framework diagram of a task processing system provided for an optional embodiment of the present invention;
[0035] Figure 3 A module diagram of a task processing device provided in an optional embodiment of the present invention. DETAILED DESCRIPTION
[0036] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.
[0037] First, the relevant terms used in the present invention are explained:
[0038] Task declaration node cluster: It is composed of task declaration nodes, provides k:v value registration function, k:v heartbeat maintenance capability, and event notification capabilities such as registration and deletion.
[0039] Task execution node: An application component that can be integrated into the task processing system, providing self-coordinating preemption capabilities during task processing and responsible for running the user's specific task program.
[0040] job: The dispatched work, also known as the work declaration event, has a certain publishing format, the publishing format is / job / key:value, key is the job name, value is the number of concurrent tasks required for the job; for example, / job / a:100 means that job a is published and this job requires 100 concurrent task instances.
[0041] task: the specific numbered task corresponding to the job. For example, the tasks corresponding to / job / a:100 are: / task / job_a / 0, / task / job_a / 1, ..., / task / job_a / 99; the task numbering range is / task / job_a / [0-99].
[0042] like Figure 1 As shown, the present invention provides a task processing method, comprising:
[0043] Step 11, obtaining at least one target work declaration event in a resource cluster through at least one task execution node in the task execution node cluster, wherein the resource cluster includes a plurality of task declaration nodes, and at least one target work declaration event is published in the resource cluster in a first format;
[0044] Step 12, determining at least one target to-be-processed task among a plurality of to-be-processed tasks of the target work declaration event;
[0045] Step 13: Execute at least one of the target tasks to be processed according to the type of the target work declaration event.
[0046] In this embodiment, the resource cluster is a task declaration node cluster, and the task declaration node cluster is composed of multiple task declaration nodes; the resource cluster can be regarded as a cluster entity, and the target work declaration event to be processed will be published in the resource cluster in a certain format, and each target work declaration event to be processed contains multiple tasks that need to be executed concurrently. Here, the publishing format of the target work declaration event is: / job / key:value, wherein key represents the task name (it can also be the name of the corresponding target declaration event), and value is the number of threads required for the task (that is, the number of tasks required to be processed by the target declaration event), such as / job / a:10, which means that the work declaration event named a contains 10 tasks that need to be processed;
[0047] The resource cluster can be used as a carrier for recording published but unprocessed work statement events and processed work statement events, and has a broadcasting capability, and can broadcast the latest status of the work statement events recorded therein in real time;
[0048] The task execution node cluster includes multiple task execution nodes, each of which can communicate with the task declaration node cluster. Each task execution node can obtain the work declaration events published in the task declaration node cluster through an event listening mechanism, and can also feed back their corresponding status to the task declaration node cluster in real time through a heartbeat maintenance mechanism;
[0049] Each task execution node in the task execution node cluster can simultaneously obtain multiple pending tasks in the same target declaration event through an event listening mechanism, so as to realize self-coordination distribution of multiple pending tasks corresponding to the target declaration event in the task declaration node cluster, thereby avoiding the traditional central node and the accompanying single point failure problem;
[0050] Furthermore, each task execution node can confirm its own target pending tasks in the current target declaration event to reduce the preemption and collision problems during the processing of pending tasks; further, based on the type of the target declaration event, each task execution node can execute the corresponding target pending tasks, thereby improving the efficiency of task processing.
[0051] In an optional embodiment of the present invention, based on the above steps 11 to 13, the following steps may also be included:
[0052] Step 14: Save the identification information of the task execution node that executes the current target task to be processed, the first name information and the first random number information of the current target task to be processed into the local task list, and send them to the resource cluster at the same time.
[0053] In this embodiment, the first name information corresponding to the target pending task is the name information of the target declaration event; after the current task execution node completes the execution of the target pending task, corresponding feedback information can be generated according to the identification information corresponding to the current task execution node, the first name information of the target pending task and the first random number information, and the feedback information is fed back to the resource cluster through the heartbeat maintenance mechanism; for all pending tasks of the same work declaration event, when they are acquired by the same task execution node, on the task execution node, all pending tasks share a heartbeat lease so as to provide real-time feedback on the processing or execution status of the tasks on the current task execution node;
[0054] It should be known that the task execution node listens to the events and maintains the heartbeat of the task declaration node cluster in real time and synchronously. For the same work declaration event, each task execution node in the task execution node cluster can obtain the pending tasks of the event. Since the speed at which each task execution node executes tasks is different, the feedback information generated by the task execution node after executing the task can be fed back to the resource cluster in real time; further, the resource cluster can broadcast the feedback information, and each task execution node in the task execution node cluster can receive the feedback information through the event listening mechanism. Based on the feedback information, other task execution nodes in the task execution node cluster can know the tasks that have been executed in the corresponding work declaration time event and the task execution nodes that execute the tasks, so as to avoid other task execution nodes from repeating the task, avoid wasting computing power, and improve the efficiency of task processing.
[0055] like Figure 2 As shown, an optional embodiment of the present invention further provides a task processing system, comprising: a resource cluster and a task execution node cluster communicating with the resource cluster through an event listening mechanism and a heartbeat maintenance mechanism; wherein the cluster resource is composed of a plurality of task declaration nodes, and the task execution node cluster is composed of a plurality of task execution nodes; a target work declaration event is published in the resource cluster in a first format, and when processing a task corresponding to the target work declaration event, specifically:
[0056] Acquire at least one target work declaration event in the resource cluster through at least one task execution node in the task execution node cluster; each task execution node can determine at least one target to-be-processed task among multiple to-be-processed tasks of the target work declaration event; and execute at least one of the target to-be-processed tasks according to the type of the target work declaration event;
[0057] The task processing system, based on the event listening mechanism and the heartbeat maintenance mechanism, can realize the autonomous distribution, collision avoidance, task drift, task dispersal, etc. of tasks between the resource cluster and the task execution node cluster, and eliminates the management nodes in the traditional sense; at the same time, the system has a specific self-healing ability and can tolerate extreme situations such as unexpected downtime of the task execution node; in the task processing system, the tasks adopt real-time listening, disorderly preemption plus caching, and registration mechanisms, which greatly reduces the task preemption collision and the task startup is efficient and fast; when any task execution node crashes or goes down, all the tasks on it will automatically drift to other nodes, and the system has a self-healing mechanism without manual intervention; at the same time, when the system can freely increase or decrease nodes, no manual processing other than node start and stop is required.
[0058] In an optional embodiment of the present invention, before obtaining at least one target work declaration event in the resource cluster, the method further includes:
[0059] Step 10a, obtaining a work declaration event list in the resource cluster in real time, and caching the work declaration event list locally;
[0060] Step 10b, obtaining a task list in the resource cluster in real time, and caching the task list locally, wherein the task list includes multiple published unexecuted tasks and executed tasks, the published unexecuted tasks correspond to first attribute information, the executed tasks correspond to second attribute information, and the executed tasks correspond to identification information of a task execution node that executes the executed tasks.
[0061] In this embodiment, before each task execution node obtains the target declaration event published in the resource cluster, through the event listening mechanism, each task execution node first queries all work declaration events in the resource cluster, establishes a corresponding work declaration event list, and caches the work declaration event list on the corresponding task execution node; preferably, after caching the work declaration event list, the current task execution node enters the listening state, keeps the work declaration events in the work declaration event list synchronized with the work declaration events in the resource cluster in real time, so as to facilitate the subsequent query and comparison of the work declaration events, and ensure the execution efficiency of the tasks to be processed in the event; here, the multiple tasks corresponding to each work declaration event in the work declaration event list can be randomly numbered to generate task numbers corresponding to the corresponding work declaration events; the work declaration event list contains processed historical work declaration events and unprocessed work declaration events;
[0062] Furthermore, through the event listening mechanism, each task execution node queries multiple tasks corresponding to each work declaration event (including historical work declaration events and unprocessed work declaration events) in the resource cluster, establishes a corresponding task list, and caches the task list on the corresponding task execution node; preferably, after caching the task list, the current task execution node enters a listening state, and keeps the tasks in the task list synchronized with the tasks corresponding to the work declaration events in the resource cluster in real time; here, the first attribute information includes: the name information of the work declaration event corresponding to the published unexecuted task (the name information is also the name information of the corresponding unprocessed task) and the number information corresponding to the unexecuted task; the second attribute information includes: the name information of the work declaration event corresponding to the executed task (the name information is also the name information of the corresponding processed task) and the number information corresponding to the executed task;
[0063] By obtaining the work declaration event list and task list, and synchronizing the work declaration events and tasks in the task declaration cluster in real time, the executable target tasks in the target declaration event obtained on the current task execution node can be determined, thereby avoiding repeated execution of tasks, avoiding wasting computing power, and improving the efficiency of task processing;
[0064] Furthermore, the task execution node may update the locally cached work declaration event list and the task list in real time according to the feedback information.
[0065] In an optional embodiment of the present invention, the above step 12 may include:
[0066] Step 121, determining the to-be-processed task interval of the target work declaration event according to the target attribute information of the target work declaration event and the historical attribute information in the resource cluster, wherein the target attribute information includes: the required number of to-be-processed tasks corresponding to the current target work declaration event, the first number information and the first name information corresponding to each required to-be-processed task, and the historical attribute information includes: the number of processed tasks corresponding to the historical work declaration event, the second number information of each processed task and the historical name information corresponding to the historical work declaration event;
[0067] Step 122: sort the required tasks to be processed in the task interval to obtain at least one target task to be processed, each target task to be processed having a first random number information.
[0068] In this embodiment, the historical attribute information can be obtained from the work declaration event list and the task list; the first name information corresponding to each required to-be-processed task in the target attribute information of the target work declaration event is the same, and is the name of the target declaration event, and the corresponding first number information can be set in sequence according to the number of required to-be-processed tasks; for example, if the number of required to-be-processed tasks for the target declaration event is 30, then the first number information corresponding to the tasks is 1, 2, 3, ..., 30 in sequence;
[0069] Here, determining the to-be-processed task interval of the target work declaration event according to the target attribute information of the target work declaration event and the historical attribute information in the resource cluster specifically includes:
[0070] When the name of the target declaration event matches the name of the historical declaration event, it means that some tasks in the target declaration event have been executed. Then, based on the required number of tasks to be processed corresponding to the target declaration event and the number of processed tasks corresponding to the historical work declaration event, the number of required tasks to be processed that have not been executed in the target declaration event can be calculated and used as the interval of tasks to be processed of the target work declaration event;
[0071] When the name of the target declaration event does not match the name of the historical declaration event, it means that the task in the target declaration event has not been executed, then confirming the number of all required pending tasks in the target declaration event is the pending task interval;
[0072] In a specific implementable example:
[0073] 1. In the resource cluster, publish a job in the format of / job / key:value, where key is the task name and value is the number of tasks; for example, / job / a:10;
[0074] 2. Each task execution node in the task execution node cluster receives the job declaration event;
[0075] 3. Each task execution node parses the key:value and pre_value of the work declaration event operation, and calculates the task number interval [m, n] that needs to be created or deleted this time; here, the pre_value value is the number of tasks that have been executed;
[0076] Furthermore, the required pending tasks in the pending task interval are randomly sorted, and then at least one target pending task with first random number information is obtained; here, the target attribute information of the target declaration work event is searched in the work declaration class table and the task list to determine the target pending task that can be executed in the target declaration event obtained on the current task execution node, thereby avoiding repeated execution of tasks, avoiding waste of computing power of the task execution node, and also improving the efficiency of task processing.
[0077] In an optional embodiment of the present invention, executing at least one of the target to-be-processed tasks according to the type of the target work declaration event may also include:
[0078] Step 130 , confirming the target task processing volume corresponding to the current task execution node according to the required number of tasks to be processed corresponding to the target work declaration event and the task processing volume threshold of the current task execution node.
[0079] Here, the task processing volume threshold of the current task execution node can be set according to the computing power of the current task execution node, and each task execution node will record the executed task processing volume once after executing each task; when the sum of the executed task processing volume and the required number of tasks to be processed corresponding to the target work declaration event is greater than the task processing volume threshold of the current task execution node, a part of the required tasks to be processed in the target work declaration event that exceeds the task processing volume threshold will not be processed (that is, not executed), and this part of the required tasks to be processed that are not processed can be randomly selected according to their corresponding task numbers, and at the same time, another part of the required tasks to be processed in the target work declaration event within the task processing volume threshold is determined as the target task processing volume; when the sum of the two is less than or equal to the task processing volume threshold of the current task execution node, all required tasks to be processed in the target work declaration event can be processed by the current task execution node;
[0080] According to the required number of tasks to be processed corresponding to the target work declaration event and the task processing volume threshold of the current task execution node, the target task processing volume corresponding to the current task execution node is confirmed, which can ensure the computing power and execution efficiency of the current task execution node when executing tasks, and at the same time avoid task execution omissions due to task accumulation.
[0081] In an optional embodiment of the present invention, when the type of the target work declaration event is creating a task, the above step 13 may include:
[0082] Step 131a, traversing the task list of the current task execution node according to the first name information and the first random number information of at least one of the target tasks to be processed, generating first registration information when the first name information and the first random number information of any target task to be processed do not match the name information and the number information in the first attribute information in the task list, and do not match the name information and the number information in the second attribute information in the task list, wherein the first registration information carries the first name information and the first random number information of the current target task to be processed and the identification information of the current task execution node;
[0083] Step 132a: Create a task execution number thread corresponding to the current target task to be processed locally according to the first registration information.
[0084] In this embodiment, the task list of the current task execution node is traversed, and when the first name information and the first random number information of any target task to be processed do not match the name information and the number information in the first attribute information in the task list, and do not match the name information and the number information in the second attribute information in the task list, it means that the current target task to be processed has not been created and executed by other task execution nodes, then the first registration information is generated according to the first name information, the first random number information of the current target task to be processed and the identification information of the current task execution node; further, according to the first registration information, a task execution number thread corresponding to the target task to be processed is created on the current task execution node; at the same time, the first registration information is fed back to the resource cluster through the heartbeat guarantee mechanism to avoid other task executions from repeatedly creating the task execution number thread corresponding to the target task to be processed;
[0085] When traversing the task list of the current task execution node, when the first name information and the first random number information of any target task to be processed match the name information and number information in the first attribute information in the task list, it means that the current target task to be processed has been created and executed by other task execution nodes. In this case, the current target task to be processed will not be processed, and the current target task to be processed will be skipped, and the traversal of the next target task to be processed will be performed.
[0086] In an optional embodiment of the present invention, when the type of the target work declaration event is to delete a task, the above step 13 may include:
[0087] Step 131b, based on the first name information and the first random number information of at least one of the target tasks to be processed, traverse the task list of the current task execution node, and when the first name information and the first random number information of any target task to be processed match the name information and the number information in the first attribute information in the task list, or match the name information and the number information in the second attribute information in the task list, generate deletion information, and the deletion information carries the first name information and the first random number information of the current target task to be processed;
[0088] Step 132b: According to the deletion information, the task execution number thread corresponding to the current target task to be processed is deleted locally.
[0089] In this embodiment, the task list of the current task execution node is traversed, and when the first name information and the first random number information of any target task to be processed match the name information and the number information in the first attribute information in the task list, or match the name information and the number information in the second attribute information in the task list, it means that the current target task to be processed has been created and executed by other task execution nodes, then deletion information is generated according to the first name information, the first random number information of the current target task to be processed and the identification information of the current task execution node; further, according to the deletion information, the task execution number thread corresponding to the target task to be processed is deleted in the local cache of the current task execution node; at the same time, the deletion information is fed back to the resource cluster through the heartbeat guarantee mechanism to avoid other tasks from executing the target task repeatedly;
[0090] When traversing the task list of the current task execution node, when the first name information and the first random number information of any target task to be processed do not match the name information and the number information in the first attribute information in the task list, or do not match the name information and the number information in the second attribute information in the task list, it means that the task execution number thread corresponding to the current target task to be processed is not on the current task execution node, then the current target task to be processed is skipped, and the next target task to be processed is traversed.
[0091] In an optional embodiment of the present invention, when the type of the target work declaration event is to delete a task, the process further includes:
[0092] Step 133b, when the identification information of the current task execution node matches the preset task execution node identification information of the current target task to be processed, locally deleting the task execution number thread corresponding to the current target task to be processed;
[0093] Step 134b, when the identification information of the current task execution node does not match the preset task execution node identification information of the current target task to be processed, traverse the task list of the current task execution node according to the first name information and the first random number information of the current target task to be processed, and when the first name information and the first random number information of any target task to be processed do not match the name information and the number information in the first attribute information in the task list, and do not match the name information and the number information in the second attribute information in the task list, generate second registration information according to the first name information and the first random number information of the current target task to be processed and the identification information of the current task execution node;
[0094] Step 135b: Send the second registration information to the resource cluster.
[0095] In this embodiment, the identification information of the preset task execution node represents the identification information corresponding to the ideal task execution node set to process the current target task to be processed; when the identification information of the current task execution node matches the preset task execution node identification information of the current target task to be processed, it means that the ideal task execution node set to process the current target task to be processed is the current task execution node, then the task execution number thread corresponding to the current target task to be processed is deleted locally at the current task execution node;
[0096] When the identification information of the current task execution node does not match the preset task execution node identification information of the current target task to be processed, it means that the ideal task execution node set to process the current target task to be processed is not the current task execution node, and it is necessary to create a task execution number thread for deleting the target task to be processed on the current task execution node to realize the drift of the target task to be processed between different task execution nodes; specifically: according to the first name information, the first random number information and the identification information of the current task execution node, generate the second registration information, and according to the second registration information, create the task execution number thread corresponding to the current target task to be processed locally on the current task execution node; at the same time, the second registration information is fed back to the resource cluster through the heartbeat maintenance mechanism, and the resource cluster feeds back the second registration information to other task execution nodes through the event listening mechanism, so that other task execution nodes no longer repeatedly process the target task to be processed.
[0097] A specific achievable example is used to illustrate the process of the above embodiment:
[0098] 1. Task execution nodes A, B, and C are currently maintaining heartbeats to the resource cluster;
[0099] 2. The current task execution node A is running / task / a / 1: A, the target pending task;
[0100] 3. Manually remove / task / a:1 on the resource cluster, or perform a range operation;
[0101] 4. Task execution nodes A, B, and C receive task deletion events from the resource cluster;
[0102] 5. When task execution node A receives the event, it detects that the node value is A, which is itself, and cancels the current task thread;
[0103] 6. When task execution node B and task execution node C receive the event, they detect that the node value is A, not themselves, and then enter the task creation phase;
[0104] 8. Task execution node B and task execution node C generate the second registration information at the same time and feed it back to the resource cluster. The first registration will succeed, and the second registration will fail. It is assumed here that task execution node C registers faster at this time.
[0105] 9. Task execution node C registers the task number on the resource cluster, starts the task execution number thread, and keeps heartbeating to the resource cluster;
[0106] 10. Task execution node B detects through the event listening mechanism that the task number already exists and abandons the task creation; then the task is successfully dispersed from task execution node A and drifts to task execution node C.
[0107] Information on the resource cluster is obtained in real time through the event listening mechanism and the heartbeat maintenance mechanism. When a task execution node crashes or goes down, all tasks on it will automatically drift to other task execution nodes. The system has a self-healing mechanism and does not require manual intervention, which improves the efficiency of task processing and avoids task congestion or failure to be processed in a timely manner.
[0108] In the above-mentioned embodiment of the present invention, at least one target work declaration event in a resource cluster is obtained through at least one task execution node in the task execution node cluster, the resource cluster includes multiple task declaration nodes, and at least one of the target work declaration events is published in the resource cluster in a first format; at least one target to-be-processed task among multiple to-be-processed tasks of the target work declaration event is determined; and at least one of the target to-be-processed tasks is executed according to the type of the target work declaration event, thereby enhancing the self-coordination capability during task processing and improving the task processing efficiency.
[0109] like Figure 3 As shown, the embodiment of the present invention further provides a task processing device 30, comprising:
[0110] An acquisition module 31 is used to acquire at least one target work declaration event in a resource cluster through at least one task execution node in a task execution node cluster, wherein the resource cluster includes a plurality of task declaration nodes, and at least one target work declaration event is published in the resource cluster in a first format;
[0111] The processing module 32 is used to determine at least one target to-be-processed task among multiple to-be-processed tasks of the target work declaration event; and execute at least one of the target to-be-processed tasks according to the type of the target work declaration event.
[0112] Optionally, before acquiring at least one target work declaration event in the resource cluster, the acquisition module 31 further includes:
[0113] Acquire a work declaration event list in the resource cluster in real time, and cache the work declaration event list locally;
[0114] A task list in the resource cluster is obtained in real time, and the task list is cached locally, wherein the task list includes multiple published unexecuted tasks and executed tasks, the published unexecuted tasks correspond to first attribute information, the executed tasks correspond to second attribute information, and the executed tasks correspond to identification information of the task execution node that executes the executed tasks.
[0115] Optionally, the processing module 32 determines at least one target to-be-processed task among a plurality of to-be-processed tasks of the target work declaration event, including:
[0116] Determine the to-be-processed task interval of the target work declaration event according to the target attribute information of the target work declaration event and the historical attribute information in the resource cluster, wherein the target attribute information includes: the required number of to-be-processed tasks corresponding to the current target work declaration event, the first number information and the first name information corresponding to each required to-be-processed task, and the historical attribute information includes: the number of processed tasks corresponding to the historical work declaration event, the second number information of each processed task and the historical name information corresponding to the historical work declaration event;
[0117] The required tasks to be processed in the task interval to be processed are sorted to obtain at least one target task to be processed, each target task to be processed having a first random number information.
[0118] Optionally, when the type of the target work declaration event is to create a task, the processing module 32 executes at least one of the target tasks to be processed according to the type of the target work declaration event, including:
[0119] According to the first name information and the first random number information of at least one of the target tasks to be processed, traverse the task list of the current task execution node, and when the first name information and the first random number information of any target task to be processed do not match the name information and the number information in the first attribute information in the task list, and do not match the name information and the number information in the second attribute information in the task list, generate first registration information, wherein the first registration information carries the first name information and the first random number information of the current target task to be processed and the identification information of the current task execution node;
[0120] According to the first registration information, a task execution number thread corresponding to the current target task to be processed is created locally.
[0121] Optionally, when the type of the target work declaration event is a deletion task, the processing module 32 executes the target to-be-processed task in the target to-be-processed task interval according to the type of the target work declaration event, including:
[0122] According to the first name information and the first random number information of at least one of the target tasks to be processed, traverse the task list of the current task execution node, and when the first name information and the first random number information of any target task to be processed match the name information and the number information in the first attribute information in the task list, or match the name information and the number information in the second attribute information in the task list, generate deletion information, and the deletion information carries the first name information and the first random number information of the current target task to be processed;
[0123] According to the deletion information, the task execution number thread corresponding to the current target task to be processed is deleted locally.
[0124] Optionally, when the type of the target work declaration event is to delete a task, the method further includes:
[0125] When the identification information of the current task execution node matches the preset task execution node identification information of the current target task to be processed, the task execution number thread corresponding to the current target task to be processed is locally deleted;
[0126] When the identification information of the current task execution node does not match the preset task execution node identification information of the current target task to be processed, traverse the task list of the current task execution node according to the first name information and the first random number information of the current target task to be processed, and when the first name information and the first random number information of any target task to be processed do not match the name information and the number information in the first attribute information in the task list, and do not match the name information and the number information in the second attribute information in the task list, generate second registration information according to the first name information and the first random number information of the current target task to be processed and the identification information of the current task execution node;
[0127] The second registration information is sent to the resource cluster.
[0128] Optionally, after the processing module 32 executes at least one of the target tasks to be processed, the processing module 32 further includes:
[0129] The identification information of the task execution node executing the current target task to be processed, the first name information and the first random number information of the current target task to be processed are saved in the local task list, and sent to the resource cluster at the same time.
[0130] It should be noted that the device is a device corresponding to the processing method of the above-mentioned task, and all implementation methods in the above-mentioned method embodiment are applicable to the embodiment of the device and can achieve the same technical effect.
[0131] The embodiment of the present invention further provides a communication device, comprising: a processor and a memory storing a computer program, wherein when the computer program is executed by the processor, the method described above is executed. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.
[0132] The embodiment of the present invention also provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute the method described above. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.
[0133] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0135] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0137] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0138] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.
[0139] In addition, it should be noted that in the apparatus and method of the present invention, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. Moreover, the steps of performing the above-mentioned series of processing can naturally be performed in chronological order according to the order of description, but it is not necessary to perform them in chronological order, and some steps can be performed in parallel or independently of each other. For those of ordinary skill in the art, it is understood that all or any steps or components of the method and apparatus of the present invention can be implemented in any computing device (including processors, storage media, etc.) or a network of computing devices in hardware, firmware, software or a combination thereof, which can be achieved by those of ordinary skill in the art using their basic programming skills after reading the description of the present invention.
[0140] Therefore, the purpose of the present invention can also be achieved by running a program or a group of programs on any computing device. The computing device can be a well-known general device. Therefore, the purpose of the present invention can also be achieved by simply providing a program product containing a program code that implements the method or device. That is to say, such a program product also constitutes the present invention, and the storage medium storing such a program product also constitutes the present invention. Obviously, the storage medium can be any well-known storage medium or any storage medium developed in the future. It should also be pointed out that in the device and method of the present invention, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. In addition, the steps of performing the above-mentioned series of processing can naturally be performed in chronological order according to the order of description, but it is not necessary to perform them in chronological order. Some steps can be performed in parallel or independently of each other.
[0141] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A task processing method, characterized in that: include: Acquire at least one target work declaration event in a resource cluster through at least one task execution node in a task execution node cluster, wherein the resource cluster includes a plurality of task declaration nodes, and at least one target work declaration event is published in the resource cluster in a first format; Determine at least one target pending task among a plurality of pending tasks of the target work declaration event; Execute at least one of the target pending tasks according to the type of the target work declaration event; Wherein, determining at least one target to-be-processed task among a plurality of to-be-processed tasks of the target work declaration event comprises: Determine the to-be-processed task interval of the target work declaration event according to the target attribute information of the target work declaration event and the historical attribute information in the resource cluster, wherein the target attribute information includes: the required number of to-be-processed tasks corresponding to the current target work declaration event, the first number information and the first name information corresponding to each required to-be-processed task, and the historical attribute information includes: the number of processed tasks corresponding to the historical work declaration event, the second number information of each processed task and the historical name information corresponding to the historical work declaration event; The required tasks to be processed in the task interval to be processed are sorted to obtain at least one target task to be processed, each target task to be processed having a first random number information.
2. The task processing method according to claim 1, characterized in that: Before obtaining at least one target work declaration event in the resource cluster, it also includes: Acquire a work declaration event list in the resource cluster in real time, and cache the work declaration event list locally; A task list in the resource cluster is obtained in real time, and the task list is cached locally, wherein the task list includes multiple published unexecuted tasks and executed tasks, the published unexecuted tasks correspond to first attribute information, the executed tasks correspond to second attribute information, and the executed tasks correspond to identification information of the task execution node that executes the executed tasks.
3. The task processing method according to claim 2, characterized in that: When the type of the target work declaration event is to create a task, executing at least one of the target tasks to be processed according to the type of the target work declaration event includes: According to the first name information and the first random number information of at least one of the target tasks to be processed, traverse the task list of the current task execution node, and when the first name information and the first random number information of any target task to be processed do not match the name information and the number information in the first attribute information in the task list, and do not match the name information and the number information in the second attribute information in the task list, generate first registration information, wherein the first registration information carries the first name information and the first random number information of the current target task to be processed and the identification information of the current task execution node; According to the first registration information, a task execution number thread corresponding to the current target task to be processed is created locally.
4. The task processing method according to claim 2, characterized in that: When the type of the target work declaration event is a deletion task, executing at least one of the target tasks to be processed according to the type of the target work declaration event includes: According to the first name information and the first random number information of at least one of the target tasks to be processed, traverse the task list of the current task execution node, and when the first name information and the first random number information of any target task to be processed match the name information and the number information in the first attribute information in the task list, or match the name information and the number information in the second attribute information in the task list, generate deletion information, wherein the deletion information carries the first name information and the first random number information of the current target task to be processed; According to the deletion information, the task execution number thread corresponding to the current target task to be processed is deleted locally.
5. The task processing method according to claim 4, characterized in that: When the type of the target work declaration event is to delete a task, it also includes: When the identification information of the current task execution node matches the preset task execution node identification information of the current target task to be processed, the task execution number thread corresponding to the current target task to be processed is locally deleted; When the identification information of the current task execution node does not match the preset task execution node identification information of the current target task to be processed, traverse the task list of the current task execution node according to the first name information and the first random number information of the current target task to be processed, and when the first name information and the first random number information of any target task to be processed do not match the name information and the number information in the first attribute information in the task list, and do not match the name information and the number information in the second attribute information in the task list, generate second registration information according to the first name information and the first random number information of the current target task to be processed and the identification information of the current task execution node; The second registration information is sent to the resource cluster.
6. The task processing method according to claim 1, characterized in that: After executing at least one of the target tasks to be processed, the method further includes: The identification information of the task execution node executing the current target task to be processed, the first name information and the first random number information of the current target task to be processed are saved in the local task list, and sent to the resource cluster at the same time.
7. A task processing device, characterized in that: include: An acquisition module, configured to acquire at least one target work declaration event in a resource cluster through at least one task execution node in a task execution node cluster, wherein the resource cluster includes a plurality of task declaration nodes, and at least one target work declaration event is published in the resource cluster in a first format; A processing module, configured to determine at least one target to-be-processed task among a plurality of to-be-processed tasks of the target work declaration event; Execute at least one of the target pending tasks according to the type of the target work declaration event; Wherein, determining at least one target to-be-processed task among a plurality of to-be-processed tasks of the target work declaration event comprises: Determine the to-be-processed task interval of the target work declaration event according to the target attribute information of the target work declaration event and the historical attribute information in the resource cluster, wherein the target attribute information includes: the required number of to-be-processed tasks corresponding to the current target work declaration event, the first number information and the first name information corresponding to each required to-be-processed task, and the historical attribute information includes: the number of processed tasks corresponding to the historical work declaration event, the second number information of each processed task and the historical name information corresponding to the historical work declaration event; The required tasks to be processed in the task interval to be processed are sorted to obtain at least one target task to be processed, each target task to be processed having a first random number information.
8. A communication device, characterized in that: include: A processor and a memory storing a computer program, wherein when the computer program is executed by the processor, the method according to any one of claims 1 to 6 is performed.
9. A computer-readable storage medium, characterized in that: Instructions are stored, and when the instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.
