A task processing method and device and related equipment

Abstract

Embodiments of the present application disclose a task processing method and device and related equipment, which are applied to a first node in a distributed processing system. After receiving a task notification of a to-be-processed task by using a watcher mechanism, the to-be-processed task can be extracted from an etcd database to a waiting queue. The to-be-processed task is generated according to a task request. Then, the to-be-processed task is processed by using a task processing function. The to-be-processed function is determined according to the to-be-processed task and various nodes of the distributed system for processing the to-be-processed task. After the to-be-processed task is processed by using the task processing function, the task state of the to-be-processed task in the etcd database is updated. In this way, the processing progress of the to-be-processed task can be recorded by using the etcd database, and the watcher mechanism of the etcd database is used to perform related notification of the task, so that each node can be queried and distributed processing is performed, thereby realizing stable and efficient processing of tasks in the distributed system.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a task processing method, apparatus, and related equipment. Background Technology

[0002] A distributed system is a loosely coupled system consisting of multiple processors interconnected through communication lines. It has four characteristics: (1) A distributed system consists of multiple computers, and the functions of the entire system are distributed across various nodes; (2) Each node in a distributed system contains its own processor and memory, and each has independent data processing capabilities. Each node can work independently and can also use shared communication lines to transmit information and coordinate task processing; (3) A large task can be divided into several subtasks, which are executed on different hosts; (4) A distributed system has a single, global process communication mechanism, which enables any process to communicate with others.

[0003] For task processing in a distributed system, since each node is independent and autonomous, to achieve coordinated processing of tasks among nodes, it is necessary to provide cluster message notification and management, task management mechanisms, etc. At the same time, it is necessary to consider that unfinished tasks can be redone after the distributed system crashes and restarts, and task records and indexes need to be persistently stored. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a task processing method that utilizes the etcd database to record tasks, thereby achieving stable and efficient task processing in a distributed system. The specific solution is as follows:

[0005] In a first aspect, this application discloses a task processing method applied to the first node in a distributed system, the method comprising:

[0006] After receiving a task notification for a pending task using the watcher mechanism, the pending task is retrieved from the etcd database and added to the waiting queue; the pending task is generated based on the task request.

[0007] The task to be processed is extracted from the waiting queue and moved to the processing queue, and then processed using a task processing function; the task processing function is determined based on the task to be processed and the node in the distributed system used to process the task to be processed.

[0008] After the pending task is processed, it is removed from the processing queue, and the task status of the pending task in the etcd database is updated; the task status of the pending task is stored in the etcd database in the form of key-value pairs.

[0009] Optionally, the tasks to be processed include synchronous tasks, and the task processing function includes synchronous task processing functions; the first node is the master node, and the task status of the tasks to be processed in the etcd database includes the synchronous task status of the tasks to be processed and the total task status of the tasks to be processed, wherein the total task status of the tasks to be processed reflects whether all the tasks to be processed have been completed.

[0010] Optionally, the task to be processed includes synchronous tasks and asynchronous tasks, the task processing function includes synchronous task processing function and asynchronous task processing function, and the task state of the task to be processed includes at least the subtask state corresponding to the first node in the asynchronous task state of the task to be processed; the step of processing the task to be processed using the task processing function includes:

[0011] The synchronization task is processed using the aforementioned synchronization task processing function;

[0012] After receiving notification that the synchronous task has been completed, the asynchronous task is processed using the asynchronous task processing function.

[0013] Modifying the task status of the pending tasks in the etcd database includes:

[0014] Update the subtask status corresponding to the first node in the asynchronous task status of the task to be processed to the completed status.

[0015] Optionally, the first node is the master node in the distributed system, and the task status of the task to be processed also includes the synchronization task status of the task to be processed. The method further includes:

[0016] After the synchronization task is completed, the synchronization task status of the pending task in the etcd database is updated to the completed status, so that each node in the distributed system can obtain the notification that the synchronization task has been completed through the watcher mechanism.

[0017] Optionally, the method further includes:

[0018] If the subtask status corresponding to each node in the asynchronous task status of the pending task is determined to be in the completed state, the total task status of the pending task in the etcd database is updated to the completed state. The total task status of the pending task reflects whether all the pending tasks have been completed.

[0019] Optionally, the method further includes:

[0020] When the second node in the distributed system crashes or the first node restarts, the task list in the etcd database is traversed to determine the task to be processed.

[0021] Optionally, the first node's memory contains a task memory structure for the task to be processed. The members of the task memory structure include the asynchronous task state and a task processing function for the task to be processed. The value of the asynchronous task state indicates whether the asynchronous task of the task to be processed has been completed. The method further includes:

[0022] When the second node in the distributed system crashes, if the task to be processed is in the waiting queue, the task processing function is updated so that the task to be processed can be processed using the updated task processing function; the updated task processing function is determined based on the task to be processed and the updated node in the distributed system used to process the task to be processed, and the updated node does not include the second node.

[0023] When the second node in the distributed system crashes, if the task to be processed is in the processing queue, the asynchronous task status in the task memory structure is set to null, a new task memory structure is generated for the task to be processed, and the task to be processed is placed in the waiting queue; the task processing function is updated so that the task to be processed can be processed using the updated task processing function.

[0024] In the new memory structure, the asynchronous task status is not empty, so that after the asynchronous task is processed, the asynchronous task status in the etcd database is updated according to the value of the asynchronous task status in the new task memory structure; the updated task processing function is determined according to the task to be processed and the node in the distributed system used to process the update of the task to be processed, and the updated node does not include the second node.

[0025] Secondly, this application discloses a task processing apparatus applied to a first node in a distributed system, the apparatus comprising:

[0026] The queue management unit is used to retrieve the task to be processed from the etcd database and add it to the waiting queue after receiving the task notification of the task to be processed using the watcher mechanism; the task to be processed is generated according to the task request.

[0027] A task processing unit is used to process the task to be processed using a task processing function; the task processing function is determined based on the task to be processed and the node in the distributed system used to process the task to be processed.

[0028] The status update unit is used to update the task status of the task to be processed in the etcd database after the task processing function has been completed; the task status of the task to be processed is stored in the etcd database in the form of key-value pairs.

[0029] The queue management unit is further configured to: extract the task to be processed from the waiting queue to the processing queue before processing the task using the task processing function; and remove the task to be processed from the processing queue after processing the task using the task processing function.

[0030] Optionally, the tasks to be processed include synchronous tasks, and the task processing function includes synchronous task processing functions; the first node is the master node, and the task status of the tasks to be processed in the etcd database includes the synchronous task status of the tasks to be processed and the total task status of the tasks to be processed, wherein the total task status of the tasks to be processed reflects whether all the tasks to be processed have been completed.

[0031] Optionally, the task to be processed includes synchronous tasks and asynchronous tasks, the task processing function includes synchronous task processing function and asynchronous task processing function, and the task state of the task to be processed includes at least the subtask state corresponding to the first node in the asynchronous task state of the task to be processed; the task processing unit includes:

[0032] A synchronization task processing unit is used to process the synchronization task using the synchronization task processing function;

[0033] An asynchronous task processing unit is used to process the asynchronous task using the asynchronous task processing function after receiving a notification that the synchronous task has been completed.

[0034] The state update unit is specifically used to update the subtask state corresponding to the first node in the asynchronous task state of the task to be processed to the completed state.

[0035] Optionally, the first node is the master node in the distributed system, and the task status of the task to be processed also includes the synchronization task status of the task to be processed. The device further includes:

[0036] The notification unit is used to update the synchronization task status of the pending task in the etcd database to the completed status after the synchronization task is processed, so that each node in the distributed system can obtain the notification that the synchronization task has been processed through the watcher mechanism.

[0037] Optionally, the device further includes:

[0038] The task status update unit is used to determine that the subtask status corresponding to each node in the asynchronous task status of the task to be processed is in a completed state, and to update the total task status of the task to be processed in the etcd database to a completed state. The total task status of the task to be processed reflects whether all the tasks to be processed have been completed.

[0039] Optionally, the device further includes:

[0040] The task determination unit is used to traverse the task list in the etcd database and determine the task to be processed from the task list when the second node in the distributed system crashes or the first node restarts, so as to process the task to be processed.

[0041] Optionally, the first node's memory contains a task memory structure for the task to be processed. The members of the task memory structure include the asynchronous task state and a task processing function for the task to be processed. The value of the asynchronous task state indicates whether the asynchronous task of the task to be processed has been completed. The device further includes:

[0042] A function update unit is used to update the task processing function when the second node in the distributed system crashes, if the task to be processed is in the waiting queue, so as to process the task to be processed using the updated task processing function; the updated task processing function is determined based on the task to be processed and the updated node in the distributed system used to process the task to be processed, and the updated node does not include the second node.

[0043] The task update unit is used to, when the second node in the distributed system crashes, if the task to be processed is in the processing queue, set the status of the asynchronous task to be processed in the task memory structure to null, generate a new task memory structure for the task to be processed, and place the task to be processed in the waiting queue; update the task processing function so as to process the task to be processed using the updated task processing function.

[0044] In the new memory structure, the asynchronous task status is not empty, so that after the asynchronous task is processed, the asynchronous task status in the etcd database is updated according to the value of the asynchronous task status in the new task memory structure; the updated task processing function is determined according to the task to be processed and the node in the distributed system used to process the update of the task to be processed, and the updated node does not include the second node.

[0045] Thirdly, this application discloses an electronic device, comprising:

[0046] Memory and processor;

[0047] The memory is used to store computer programs;

[0048] The processor is used to execute the computer program to implement the aforementioned disclosed task processing method.

[0049] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned task processing method.

[0050] This application provides a task processing method, apparatus, and related equipment applied to the first node of a distributed processing system. After receiving a task notification for a task to be processed using a watcher mechanism, the task can be retrieved from the etcd database and added to a waiting queue. The task to be processed is generated based on a task request, and then retrieved from the waiting queue to a processing queue. A task processing function is used to process the task, which is determined based on the task and various nodes in the distributed system used to process the task. After the task processing function completes processing, the task is removed from the processing queue, and the task status of the task to be processed in the etcd database is updated. The task status can be stored in the etcd database in the form of key-value pairs. This allows the etcd database to record the processing progress of the task to be processed and to use the watcher mechanism of the etcd database to notify relevant tasks, facilitating querying by various nodes and distributed processing. Even if the distributed system crashes and restarts, unfinished tasks can still be executed smoothly, thereby achieving stable and efficient task processing in the distributed system. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0052] Figure 1 A flowchart illustrating a task processing method provided in an embodiment of this application;

[0053] Figure 2 A schematic diagram illustrating a task processing procedure provided in an embodiment of this application;

[0054] Figure 3 A structural block diagram of a task processing device provided in an embodiment of this application;

[0055] Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in this application. Detailed Implementation

[0056] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0057] As explained in the background section, for task processing in a distributed system, since each node is independent and autonomous, in order to achieve coordinated processing of tasks among the nodes, it is necessary to provide cluster message notification and management, task management mechanisms, etc. At the same time, considering that the unfinished tasks can be redone after the distributed system crashes and restarts, it is necessary to persistently store and record indexes of tasks.

[0058] Based on the above technical problems, this application provides a task processing method, apparatus, and related equipment, applied to the first node of a distributed processing system. After receiving a task notification for a task to be processed using a watcher mechanism, the task to be processed can be extracted from the etcd database and added to a waiting queue. The task to be processed is generated according to the task request, and then extracted from the waiting queue to a processing queue. The task to be processed is processed using a task processing function, which is determined based on the task to be processed and various nodes in the distributed system used to process the task. After the task to be processed is completed using the task processing function, the task to be processed is removed from the processing queue, and the task status of the task to be processed in the etcd database is updated. The task status of the task to be processed can be stored in the etcd database in the form of key-value pairs. This allows the etcd database to record the processing progress of the task to be processed and to use the watcher mechanism of the etcd database to notify the relevant tasks, facilitating querying by each node and distributed processing. Even if the distributed system crashes and restarts, unfinished tasks can still be executed smoothly, thereby achieving stable and efficient task processing in the distributed system.

[0059] The following description, in conjunction with the accompanying drawings, details the specific implementation of the task processing method, apparatus, and related equipment provided in the embodiments of this application.

[0060] refer to Figure 1 The diagram shown is a flowchart of a task processing method provided in an embodiment of this application. A distributed system can have multiple nodes. This method can be applied to the first node in the distributed system and includes the following steps:

[0061] S101: After receiving the task notification of the task to be processed using the watcher mechanism, the task to be processed is retrieved from the etcd database and added to the waiting queue.

[0062] In this embodiment of the application, the distributed system may include multiple nodes. Specifically, it may include one master node and multiple slave nodes. Each node can be used to jointly process the task to be processed. The master node can be used to communicate with external devices. The first node can be either a master node or a slave node.

[0063] etcd is a highly available key-value store designed to reliably and quickly store critical data and provide data access. It achieves reliable distributed collaboration through distributed locks, leader election, and write barriers. Essentially a distributed key-value store, etcd meets the needs of record indexing and also features message subscription and registration mechanisms. Distributed notification and collaboration utilize etcd's watcher mechanism (the publish-subscribe mechanism in etcd). Through registration and asynchronous notification mechanisms, it enables notification and coordination between different systems in a distributed environment, thereby allowing for real-time processing of data changes.

[0064] Therefore, in this embodiment, the etcd database can be used to store the status of tasks, and the watcher mechanism of etcd can be used to send task notifications, so as to realize the distribution, synchronization and coordination of tasks in the distributed system. At the same time, it can query the status and progress of tasks, and can continue unfinished tasks even after a node or cluster crashes. Thus, it provides a stable, efficient and referable implementation method for task processing architecture in distributed systems.

[0065] In this embodiment, the task to be processed is generated based on a task request. As a server, the distributed system supports access requests via command line or RESTful API (RESTful web interface). Therefore, there are external tasks originating from outside the distributed system, which are access requests from command line or RESTful API. Simultaneously, the distributed system also contains tasks that need to be executed by nodes during its operation; these are called internal tasks, and they are related to the system's business logic. The task to be processed can be either an external task or an internal task. The task to be processed can be generated by the node that receives the task request; this node can be called a proxy node. The proxy node can be a master node or a slave node, and can be the first node or any other node besides the first node.

[0066] Tasks to be processed can include synchronous tasks. Synchronous tasks are the parts of a task to be processed that need to be processed synchronously. They are commonly seen in non-time-consuming and non-blocking tasks such as memory processing and databases. In this case, the task processing function includes the synchronous task processing function. If the task to be processed only includes synchronous tasks, then the task to be processed is considered complete once the synchronous tasks are completed.

[0067] Tasks to be processed can include synchronous tasks and asynchronous tasks. Asynchronous tasks are those that need to be processed asynchronously and are often seen in time-consuming blocking tasks such as data reading and writing. In this case, the task processing function can include synchronous task processing functions and asynchronous task processing functions. The task to be processed is considered complete only after both synchronous and asynchronous tasks are completed.

[0068] After a task is generated, it can be stored in the etcd database with a specific prefix. Nodes in the distributed system can retrieve the task from the etcd database into their local waiting queue and establish a task memory structure related to the task in their local memory. This task memory structure can include multiple members, such as the unique identifier of the task, processing status, and basic information. The processing status can include synchronous task status, asynchronous task status, and overall task status. Referring to Table 1, an example of a Task memory structure provided in this application embodiment is shown.

[0069] The process involves several steps: A Task is assigned a unique, auto-incrementing sequence number (task_id) by etcd. The Task's overall state (task_state) can be set to the initial state (INIT), indicating no processing. This state signifies whether the task is fully completed. Each Task has a key that needs to be updated after a synchronous task is completed; this key is called the synchronous task state (task_sync_key), indicating whether the synchronous tasks within the Task are complete. The Task's processing function is set, which can include synchronous task processing functions (task_sync_process) and asynchronous processing functions (task_async_process). Each node uses different processing functions to process its tasks. The Task also has a key for notification after an asynchronous task is completed; this key is called the asynchronous task state (task_key), indicating whether the asynchronous tasks within the Task are complete. The task_key includes, but is not limited to, the task_id. The asynchronous task state value can include multiple fields representing the subtask states of each node. Typically, each node has a subset of these fields. After a node completes processing its subtasks, it updates the corresponding fields to indicate that the node's subtask state is complete.

[0070] Table 1 Example of Task Memory Structure

[0071]

[0072]

[0073] After a task is generated, it can be checked whether there is a synchronous task. If there is, the watcher will notify the user based on the task_sync_key when the synchronous task is completed. Otherwise, no notification will be sent when the synchronous task is completed.

[0074] Once a task is generated, it will be updated in the etcd database using a specific prefix. Nodes in the distributed system will receive notifications of the task's generation through etcd's watcher mechanism. After the task is processed, the nodes in the distributed system will also update the corresponding status value in the etcd database based on the task status parameters in their local task memory structure. There is a corresponding relationship between the processing status in the local task memory structure and the task status in the etcd database.

[0075] In this embodiment, after receiving a task notification for a pending task using the watcher mechanism, the first node in the distributed system can retrieve the pending task from the etcd database and add it to its local waiting queue, then notify the task processing worker thread to process the pending task. Of course, other nodes in the distributed system can also simultaneously receive task notifications for pending tasks using the watcher mechanism, retrieve the pending task from the etcd database and add it to their local waiting queues, and process the pending task. By having multiple nodes process the pending subtasks, parallel processing of pending tasks in the distributed system can be achieved.

[0076] S102, retrieve the task to be processed from the waiting queue to the processing queue, and use the task processing function to process the task to be processed.

[0077] In this embodiment, after obtaining the task to be processed, a task processing function can be used to process the task. Different nodes can correspond to different task processing functions, thus allowing for different processing methods for the task. The task processing function is determined based on the task to be processed and the nodes in the distributed system used to process it. This is equivalent to dividing the task to be processed into multiple parallel operations based on the operations it needs to perform, and these operations are distributed to different nodes in the form of task processing functions. Before processing the task, it can be retrieved from the waiting queue to the processing queue for further processing.

[0078] When the pending tasks include synchronous tasks, the pending tasks are processed using task processing functions. Specifically, synchronous tasks are processed using synchronous task functions. At this point, the processing of synchronous tasks is complete, indicating that the entire pending task has been processed.

[0079] When the pending tasks include synchronous and asynchronous tasks, the task processing function is used to process the pending tasks. Specifically, the synchronous task processing function is used to process the synchronous task, and after receiving the notification that the synchronous task has been completed, the asynchronous task processing function is used to process the asynchronous task, thereby completing the processing of the pending tasks.

[0080] In practice, after obtaining the tasks to be processed, it can be determined whether there are synchronous tasks among them. If so, the synchronous tasks are processed first; otherwise, it indicates that the tasks to be processed are non-compliant and are not processed. Specifically, this can be done by checking whether the tasks to be processed have defined a synchronous task processing function (task_sync_process). If so, it means that there are synchronous tasks among the tasks to be processed, and the synchronous tasks can be processed based on task_sync_process.

[0081] When the primary node is the master node in the distributed system, after completing the synchronization task of the pending task, it can update the synchronization task status (task_sync_key) of the pending task in the etcd database, for example, to a completed status. The etcd database, based on a watcher mechanism, can then notify each node of the updated synchronization task status. In other words, the master node can trigger a synchronization task completion notification upon completion of its own synchronization task, allowing external devices to initiate other operations based on this notification without waiting for the asynchronous task to finish. If the synchronization task of the pending task fails, the synchronization task status in the etcd database is updated to failure. Both failure and completion statuses are used as the value of task_sync_key. After the synchronization task status of the pending task is set to failure, the pending task can be removed from the in-progress task queue.

[0082] After completing the synchronous tasks of the pending tasks, the asynchronous tasks can be processed. Specifically, asynchronous task processing can begin after receiving notification that the synchronous task has been completed. In practice, the asynchronous tasks can be processed using the asynchronous task processing function (task_async_process).

[0083] S103: After the pending task is completed, remove the pending task from the processing queue and update the task status of the pending task in the etcd database.

[0084] After an asynchronous task is completed, it indicates that the pending task has finished processing. At this point, the pending task can be removed from the processing queue, and its task status in the etcd database can be updated. The task status of pending tasks can be stored in the etcd database as key-value pairs. The first node can update the values ​​in the key-value pairs in the etcd database based on the values ​​of the members in the local task memory structure.

[0085] When the pending tasks include only synchronous tasks and not asynchronous tasks, the completion of a synchronous task means the completion of the pending tasks. If the first node is the master node, the task status of the pending tasks can include the synchronous task status and the overall task status. Therefore, after the synchronous task is completed, the synchronous task status (task_sync_process) of the pending tasks can be updated, for example, to the completed state, and the overall task status (task_state) of the pending tasks can also be updated, for example, to the completed state.

[0086] When the pending tasks consist only of synchronous and asynchronous tasks, the completion of an asynchronous task signifies the completion of the pending task. The task status of a pending task must include at least the subtask status corresponding to the first node in the asynchronous task status. Therefore, modifying the task status of the pending task in the etcd database can be done by updating the subtask status (task_key) corresponding to the first node in the asynchronous task status to "completed." If the asynchronous task fails, the asynchronous task status in the etcd database is updated to "failed," with both the failed and completed statuses used as the value of task_key. After the first node completes processing the asynchronous task, its identifier can be written to done_list to indicate that the first node has completed the pending task. If the asynchronous task fails, done_list is not updated.

[0087] After processing all pending tasks, it can be determined whether the subtask statuses corresponding to each node in the asynchronous task status of the pending task are all in the "completed" state. If so, the entire pending task is complete, and the overall task status of the pending task (task_state) in the etcd database can be updated to "DONE". Otherwise, the task status of the pending task is not updated. Specifically, "done_list" is used to store nodes that have completed pending tasks. Since the subtask statuses corresponding to nodes in "done_list" are in the "completed" state, it can be determined whether the nodes in "done_list" represent all nodes that have processed the pending task, thus confirming whether the subtask statuses corresponding to each node in the asynchronous task are all in the "completed" state.

[0088] While processing tasks, nodes can update the progress of tasks in etcd in real time, thereby enabling the query of the progress and status of tasks and facilitating distributed task processing.

[0089] refer to Figure 2 The diagram shown is a schematic of a task processing process provided in an embodiment of this application. It takes three nodes in a distributed system as an example for illustration. The first node can act as a proxy node, i.e., used for task generation. The first node can also act as the master node, used for task synchronization completion notification. After receiving a task request, the first node can use its internal task generation device to assemble the external request into a task and place it in the etcd database. This triggers the etcd database to issue a task notification using the watcher mechanism. Subsequently, the other three nodes receive the task notification, retrieve the task from the etcd database into their local queues, and process the task using their respective task processing devices. During task processing, synchronous task processing is performed first. After the first node (i.e., the master node) completes the task synchronization task, it can update the synchronization task status of the task in the etcd database to trigger the etcd database to issue a task synchronization completion notification using the watcher mechanism. Then, each node processes the task asynchronously. After each node completes its asynchronous task, it updates the sub-states in its corresponding asynchronous task status until all nodes have completed their asynchronous tasks. At this point, the overall task status of the task can be updated to the completed state.

[0090] In this embodiment, if a node crashes during the processing of a task, the normal nodes in the distributed system can handle the tasks originally belonging to the crashed node. Alternatively, if the distributed system requires a global restart, each node needs to handle any unfinished tasks after restarting.

[0091] Specifically, when the second node in a distributed system fails, some tasks originally belonging to the second node may be reassigned to the first node. The first node can then iterate through the task list in the etcd database, identify the tasks to be processed, and begin processing them. The processing method for these tasks can be found in steps S101-S103.

[0092] Specifically, when the first node restarts, some pending tasks belonging to the first node may be lost. The first node can then iterate through the task list in the etcd database to identify and process the pending tasks. The processing method for these pending tasks can be found in steps S101-S103.

[0093] Specifically, when the second node in the distributed system fails, if the tasks to be processed are in the waiting queue (task_pending_list), the task processing function is updated to handle them. The updated task processing function is determined based on the tasks to be processed and the updated nodes in the distributed system used to process them. The updated nodes do not include the second node. In other words, task processing functions can be reallocated based on the normal nodes to ensure that the tasks to be processed are completed by these normal nodes. The updated nodes can be located in the done_list of tasks to be processed. The task processing is complete when all nodes in the done_list have finished processing their tasks.

[0094] Specifically, the task memory structure in the memory of the first node can include multiple members, which can include the asynchronous task status and task processing function of the task to be processed. The value of the asynchronous task status indicates whether the asynchronous task to be processed has been completed. So, when the second node in the distributed system crashes, if the task to be processed is in the processing queue (task_processing_list), the asynchronous task status (task_key) of the task to be processed in the local task memory structure can be set to empty. This prevents the task to be processed from updating the corresponding asynchronous task status in etcd based on the value of the asynchronous task status in the local task memory structure when it is completed. At the same time, a new task memory structure can be generated for the task to be processed, and the task to be processed can be placed in the waiting queue (task_pending_list) so that the task to be processed can be reprocessed based on the new task memory structure. The asynchronous task status in the new memory structure is not empty, so after the asynchronous task is processed, the asynchronous task status in the etcd database can be updated based on the value of the asynchronous task status in the new task memory structure. In addition, the task processing function can be updated to process the task to be processed using the updated task processing function. The updated task processing function is determined based on the task to be processed and the updated node used to process the task in the distributed system. The updated node does not include the second node.

[0095] In practical implementation, the processing of tasks can have different processing modes (epochs), and different processing modes can have different processing nodes and different task processing functions. Each processing mode can have all normal nodes in the distributed system as processing nodes for the task; specifically, each processing mode can have a node list. To determine whether the nodes in `done_list` are all nodes processing the task, we can check if the nodes in `done_list` are all nodes in the node list for that mode.

[0096] For example, Mode 1 corresponds to the first node, the second node, and the third node, while Mode 2 corresponds to the first node and the third node. In Mode 1, the task to be processed is handled using the first, second, and third nodes. The task to be processed has a task state under Mode 1, such as synchronous task state, asynchronous task state, overall task state, and done_list. The asynchronous task state includes the sub-task states corresponding to the three nodes. When done_list contains three nodes, it indicates that the task to be processed has been completed. When the second node fails, the processing mode of the task to be processed switches from Mode 1 to Mode 2, that is, the first and third nodes begin to handle the task to be processed. The first and third nodes use the updated task processing functions to process the task to be processed. Mode 2 has a task state different from that of Mode 1, representing the processing state of the first and third nodes for the task to be processed. Mode 2 can have synchronous task state, asynchronous task state, overall task state, and done_list. The asynchronous task state includes the sub-task states corresponding to the two nodes. When done_list contains two nodes, it indicates that the task to be processed has been completed.

[0097] This application provides a task processing method applied to the first node of a distributed processing system. After receiving a task notification for a task to be processed using a watcher mechanism, the method retrieves the task from the etcd database into a waiting queue. The task to be processed is generated based on the task request and then retrieved from the waiting queue into a processing queue. A task processing function is used to process the task, which is determined based on the task and the various nodes in the distributed system used to process the task. After the task processing function completes processing, the task is removed from the processing queue, and the task status of the task to be processed in the etcd database is updated. The task status can be stored in the etcd database as key-value pairs. This allows the etcd database to record the processing progress of the task and uses the watcher mechanism of the etcd database to notify relevant tasks, facilitating querying by various nodes and distributed processing. Even if the distributed system crashes and restarts, unfinished tasks can still be executed smoothly, thereby achieving stable and efficient task processing in the distributed system.

[0098] Based on the above task processing method, this application embodiment also provides a task processing device, referencing... Figure 3 The diagram shown is a structural block diagram of a task processing device provided in an embodiment of this application, comprising:

[0099] The queue management unit 110 is used to process the task to be processed using a task processing function; the task processing function is determined based on the task to be processed and the node in the distributed system used to process the task to be processed.

[0100] The task processing unit 120 is used to process the task to be processed using a task processing function; the task processing function is determined based on the task to be processed and the node in the distributed system used to process the task to be processed.

[0101] The status update unit 130 is used to update the task status of the task to be processed in the etcd database after the task processing function has been completed; the task status of the task to be processed is stored in the etcd database in the form of key-value pairs.

[0102] The queue management unit is further configured to: extract the task to be processed from the waiting queue to the processing queue before processing the task using the task processing function; and remove the task to be processed from the processing queue after processing the task using the task processing function.

[0103] Optionally, the tasks to be processed include synchronous tasks, and the task processing function includes synchronous task processing functions; the first node is the master node, and the task status of the tasks to be processed in the etcd database includes the synchronous task status of the tasks to be processed and the total task status of the tasks to be processed, wherein the total task status of the tasks to be processed reflects whether all the tasks to be processed have been completed.

[0104] Optionally, the task to be processed includes synchronous tasks and asynchronous tasks, the task processing function includes synchronous task processing function and asynchronous task processing function, and the task state of the task to be processed includes at least the subtask state corresponding to the first node in the asynchronous task state of the task to be processed; the task processing unit includes:

[0105] A synchronization task processing unit is used to process the synchronization task using the synchronization task processing function;

[0106] An asynchronous task processing unit is used to process the asynchronous task using the asynchronous task processing function after receiving a notification that the synchronous task has been completed.

[0107] The state update unit is specifically used to update the subtask state corresponding to the first node in the asynchronous task state of the task to be processed to the completed state.

[0108] Optionally, the first node is the master node in the distributed system, and the task status of the task to be processed also includes the synchronization task status of the task to be processed. The device further includes:

[0109] The notification unit is used to update the synchronization task status of the pending task in the etcd database to the completed status after the synchronization task is processed, so that each node in the distributed system can obtain the notification that the synchronization task has been processed through the watcher mechanism.

[0110] Optionally, the device further includes:

[0111] The task status update unit is used to determine that the subtask status corresponding to each node in the asynchronous task status of the task to be processed is in a completed state, and to update the total task status of the task to be processed in the etcd database to a completed state. The total task status of the task to be processed reflects whether all the tasks to be processed have been completed.

[0112] Optionally, the device further includes:

[0113] The task determination unit is used to traverse the task list in the etcd database and determine the task to be processed from the task list when the second node in the distributed system crashes or the first node restarts, so as to process the task to be processed.

[0114] Optionally, the first node's memory contains a task memory structure for the task to be processed. The members of the task memory structure include the asynchronous task state and a task processing function for the task to be processed. The value of the asynchronous task state indicates whether the asynchronous task of the task to be processed has been completed. The device further includes:

[0115] A function update unit is used to update the task processing function when the second node in the distributed system crashes, if the task to be processed is in the waiting queue, so as to process the task to be processed using the updated task processing function; the updated task processing function is determined based on the task to be processed and the updated node in the distributed system used to process the task to be processed, and the updated node does not include the second node.

[0116] The task update unit is used to, when the second node in the distributed system crashes, if the task to be processed is in the processing queue, set the state of the asynchronous task to be processed in the task memory structure to null, generate a new task memory structure for the task to be processed, and place the task to be processed in the waiting queue; update the task processing function so as to process the task to be processed using the updated task processing function.

[0117] In the new memory structure, the asynchronous task status is not empty, so that after the asynchronous task is processed, the asynchronous task status in the etcd database is updated according to the value of the asynchronous task status in the new task memory structure; the updated task processing function is determined according to the task to be processed and the node in the distributed system used to process the update of the task to be processed, and the updated node does not include the second node.

[0118] This application provides a task processing device applied to the first node of a distributed processing system. After receiving a task notification for a task to be processed using a watcher mechanism, the device can retrieve the task from the etcd database and add it to a waiting queue. The task to be processed is generated based on the task request and then retrieved from the waiting queue to a processing queue. A task processing function is used to process the task, which is determined based on the task and the various nodes in the distributed system used to process the task. After the task processing function completes processing, the task is removed from the processing queue, and the task status of the task to be processed in the etcd database is updated. The task status can be stored in the etcd database as key-value pairs. This allows the etcd database to record the processing progress of the task and uses the watcher mechanism of the etcd database to notify relevant tasks, facilitating queries by various nodes and distributed processing. Even if the distributed system crashes and restarts, unfinished tasks can still be executed smoothly, thereby achieving stable and efficient task processing in the distributed system.

[0119] See Figure 4 The diagram shown is a structural schematic of an electronic device 20 provided in an embodiment of this application. The electronic device 20 can implement the task processing method steps disclosed in the foregoing embodiments.

[0120] Typically, the electronic device 20 in this embodiment includes a processor 21 and a memory 22.

[0121] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware: DSP (digital signal processing), FPGA (field-programmable gate array), or PLA (programmable logic array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (central processing unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (graphics processing unit), which is responsible for rendering and drawing images required for display on the screen. In some embodiments, the processor 21 may include an AI (artificial intelligence) processor, which is used to handle computational operations related to machine learning.

[0122] The memory 22 may include one or more computer-readable storage media, which may be non-transitory. The memory 22 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 22 is used to store at least the following computer program 221, which, after being loaded and executed by the processor 21, is capable of implementing the task processing method steps disclosed in any of the foregoing embodiments.

[0123] In some embodiments, the electronic device 20 may further include a display screen 23, an input / output interface 24, a communication interface 25, a sensor 26, a power supply 27, and a communication bus 28.

[0124] Those skilled in the art will understand. Figure 3 The structure shown does not constitute a limitation on the electronic device 20 and may include more or fewer components than shown.

[0125] Furthermore, embodiments of this application also disclose a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the task processing method disclosed in any of the foregoing embodiments.

[0126] The specific process of the above task processing method can be found in the relevant content disclosed in the foregoing embodiments, and will not be repeated here.

[0127] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system and device embodiments are basically similar to the method embodiments, so they are described relatively simply; relevant parts can be referred to the descriptions of the method embodiments.

[0128] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0129] The term "first" mentioned in this application embodiment is used only for name identification and does not represent the first in order. The same rule applies to "second," etc.

[0130] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A task processing method, characterized in that, The method, applied to the first node in a distributed system, includes: After receiving a task notification for a pending task using the watcher mechanism, the pending task is retrieved from the etcd database and added to the waiting queue; the pending task is generated according to the task request. The task to be processed is extracted from the waiting queue and moved to the processing queue, and then processed using a task processing function; the task processing function is determined based on the task to be processed and the node in the distributed system used to process the task to be processed. After the pending task is processed, the pending task is removed from the processing queue, and the task status of the pending task in the etcd database is updated; the task status of the pending task is stored in the etcd database in the form of key-value pairs. When the second node in the distributed system crashes, if the task to be processed is in the waiting queue, the task processing function is updated so that the task to be processed can be processed using the updated task processing function; the updated task processing function is determined based on the task to be processed and the updated node in the distributed system used to process the task to be processed, and the updated node does not include the second node. The first node's memory contains a task memory structure for the task to be processed. Members of the task memory structure include the asynchronous task state and a task processing function for the task to be processed. The value of the asynchronous task state indicates whether the asynchronous task of the task to be processed has been completed. The method further includes: When the second node in the distributed system crashes, if the task to be processed is in the processing queue, the asynchronous task status of the task to be processed in the task memory structure is set to null, a new task memory structure is generated for the task to be processed, and the task to be processed is placed in the waiting queue; the task processing function is updated so that the task to be processed can be processed using the updated task processing function. In the new memory structure, the asynchronous task status is not empty, so that after the asynchronous task is processed, the asynchronous task status in the etcd database is updated according to the value of the asynchronous task status in the new task memory structure; the updated task processing function is determined according to the task to be processed and the node in the distributed system used to process the update of the task to be processed, and the updated node does not include the second node.

2. The method according to claim 1, characterized in that, The tasks to be processed include synchronous tasks, and the task processing function includes synchronous task processing functions; the first node is the master node, and the task status of the tasks to be processed in the etcd database includes the synchronous task status of the tasks to be processed and the total task status of the tasks to be processed, wherein the total task status of the tasks to be processed reflects whether all the tasks to be processed have been completed.

3. The method according to claim 1, characterized in that, The tasks to be processed include synchronous tasks and asynchronous tasks, the task processing functions include synchronous task processing functions and asynchronous task processing functions, and the task status of the tasks to be processed includes at least the subtask status corresponding to the first node in the asynchronous task status of the tasks to be processed. The process of processing the task to be processed using the task processing function includes: The synchronization task is processed using the aforementioned synchronization task processing function; After receiving notification that the synchronous task has been completed, the asynchronous task is processed using the asynchronous task processing function. Modifying the task status of the pending tasks in the etcd database includes: Update the subtask status corresponding to the first node in the asynchronous task status of the task to be processed to the completed status.

4. The method according to claim 3, characterized in that, The first node is the master node in the distributed system, and the task status of the task to be processed also includes the synchronization task status of the task to be processed. The method further includes: After the synchronization task is completed, the synchronization task status of the pending task in the etcd database is updated to the completed status, so that each node in the distributed system can obtain the notification that the synchronization task has been completed through the watcher mechanism.

5. The method according to claim 3 or 4, characterized in that, The method further includes: If the subtask status corresponding to each node in the asynchronous task status of the pending task is determined to be in the completed state, the total task status of the pending task in the etcd database is updated to the completed state. The total task status of the pending task reflects whether all the pending tasks have been completed.

6. The method according to any one of claims 1-4, characterized in that, The method further includes: When the second node in the distributed system crashes or the first node restarts, the task list in the etcd database is traversed to determine the task to be processed.

7. A task processing device, characterized in that, The device, used as the first node in a distributed system, includes: The queue management unit is used to retrieve the task to be processed from the etcd database and add it to the waiting queue after receiving the task notification of the task to be processed using the watcher mechanism; the task to be processed is generated according to the task request. A task processing unit is used to process the task to be processed using a task processing function; the task processing function is determined based on the task to be processed and the node in the distributed system used to process the task to be processed. The status update unit is used to update the task status of the task to be processed in the etcd database after the task processing function has been completed; the task status of the task to be processed is stored in the etcd database in the form of key-value pairs. The queue management unit is further configured to: extract the task to be processed from the waiting queue to the processing queue before processing the task using the task processing function; and remove the task to be processed from the processing queue after processing the task using the task processing function. A function update unit is used to update the task processing function when the second node in the distributed system crashes, if the task to be processed is in the waiting queue, so as to process the task to be processed using the updated task processing function; the updated task processing function is determined based on the task to be processed and the updated node in the distributed system used to process the task to be processed, and the updated node does not include the second node. The first node's memory contains a task memory structure for the task to be processed. Members of the task memory structure include the asynchronous task state and a task processing function for the task to be processed. The value of the asynchronous task state indicates whether the asynchronous task of the task to be processed has been completed. The device further includes: The task update unit is used to, when the second node in the distributed system crashes, if the task to be processed is in the processing queue, set the status of the asynchronous task to be processed in the task memory structure to null, generate a new task memory structure for the task to be processed, and place the task to be processed in the waiting queue; update the task processing function so as to process the task to be processed using the updated task processing function. In the new memory structure, the asynchronous task status is not empty, so that after the asynchronous task is processed, the asynchronous task status in the etcd database is updated according to the value of the asynchronous task status in the new task memory structure; the updated task processing function is determined according to the task to be processed and the node in the distributed system used to process the update of the task to be processed, and the updated node does not include the second node.

8. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the task processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the task processing method as described in any one of claims 1 to 6.

Images