Method and device for processing distributed data

Abstract

The invention discloses a method and device for processing distributed data, and relates to the technical field of big data. A specific embodiment of the method comprises the steps of obtaining a plurality of data fragments of to-be-processed data, constructing corresponding tasks for the data fragments, and storing the tasks in a task control table; determining a target high-availability strategy matched with the to-be-processed data from a plurality of high-availability distributed processing strategies; and preempting a plurality of target tasks of which the task states are to be processed in the task control table by utilizing the distributed nodes and the target high-availability strategy, processing corresponding data fragments through the plurality of target tasks, and monitoring task running conditions of the distributed nodes and executing task processing operation by utilizing a monitoring strategy corresponding to the target high-availability strategy. According to the embodiment of the invention, the scene adaptability, flexibility and expansibility of distributed data processing are improved, and the efficiency and high availability of processing the distributed data are improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method and device for processing distributed data. Background Art

[0002] With the widespread use of Internet application systems, the requirements for system processing capabilities are becoming increasingly higher. For application scenarios that require streaming data processing (such as processing financial data), there are higher requirements for the real-time and reliability of the system's processing of streaming data.

[0003] Currently, a distributed structure is usually used to process streaming data, which mainly connects multiple computing nodes through a network and controls the data processing of each computing node through a control center to complete the preset data processing tasks in parallel. The existing method will cause the entire system to process abnormally if any computing node has an abnormality, resulting in poor system fault tolerance. In addition, since the number of computing nodes is fixed, it leads to poor flexibility such as lack of resources or waste of resources. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method and device for processing distributed data, which can obtain multiple data slices of data to be processed, construct corresponding tasks for the data slices, and store the tasks in a task control table; determine a target high-availability strategy that matches the data to be processed from a variety of high-availability distributed processing strategies; use distributed nodes and target high-availability strategies to seize multiple target tasks in the task control table whose task status is to be processed, process the corresponding data slices through multiple target tasks, and use the abnormal monitoring strategy corresponding to the target high-availability strategy to monitor the task running status of the distributed nodes and perform task processing operations accordingly. The embodiment of the present invention improves the scenario adaptability, flexibility and scalability of distributed data processing, and improves the efficiency and high availability of processing distributed data.

[0005] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, a method for processing distributed data is provided, which is characterized by comprising: obtaining multiple data slices of the data to be processed, constructing a corresponding task for each of the data slices, and storing the tasks in a task control table; determining a target high-availability strategy that matches the data to be processed from a plurality of high-availability distributed processing strategies; utilizing distributed nodes and the target high-availability strategy to preempt multiple target tasks whose task status is to be processed in the task control table, and processing corresponding data slices through the multiple target tasks; utilizing the abnormal monitoring strategy corresponding to the target high-availability strategy to monitor the task running status of the distributed nodes and perform corresponding task processing operations according to the task running status.

[0006] Optionally, in the case where the target high availability strategy is a preemptive high availability strategy, the use of distributed nodes and the target high availability strategy to preempt the target task whose task status is pending in the task control table includes: for any distributed node, starting the main control thread of the node, and using the main control thread to generate multiple processing threads for processing tasks according to the number of shards of the data shards; using the main control thread to obtain each of the target tasks to be processed, and storing the target tasks to local storage, and performing a reordering operation on each target task in the local storage, so that multiple processing threads perform the operation of preempting the target task based on the reordered tasks.

[0007] Optionally, in the case where the target high availability strategy is a preemptive high availability strategy, the abnormal monitoring strategy corresponding to the target high availability strategy is used to monitor the task running status of the distributed nodes and perform corresponding task processing operations according to the task running status, including: registering multiple distributed nodes in a monitoring center; using the monitoring center to perform the following operations based on the abnormal monitoring strategy: monitoring each of the distributed nodes to create different heartbeat files at set time intervals; when it is monitored that the heartbeat file of any distributed node is not updated at a set number of set time intervals, determining that there is an abnormality in the task running status of the distributed node; for the distributed node with abnormal task running status, obtaining the current task processed by the thread of the abnormal distributed node, and changing the task status of the current task to pending in the task control table.

[0008] Optionally, for the case where the target high availability strategy is a master-standby high availability strategy, the use of distributed nodes and the target high availability strategy to preempt the target task whose task status is pending in the task control table includes: determining multiple master nodes from multiple distributed nodes; obtaining the task identifier of the pending task and the number of master nodes; determining each pending task identifier processed by each master node based on a first calculation relationship between the number of nodes and the task identifier; using the main control thread of each master node to construct multiple processing threads for processing tasks, and using the processing threads to preempt the target task corresponding to the pending task identifier in the task control table.

[0009] Optionally, determining multiple master nodes from the multiple distributed nodes includes: the multiple distributed nodes connect to the distributed coordination system at startup, and create a unique directory identifier in the set storage location of the distributed coordination system, and the distributed node that successfully creates the unique target identifier is used as the master node; using the abnormal monitoring strategy corresponding to the target high availability strategy to monitor the task running status of the distributed nodes and perform corresponding task processing operations according to the task running status, including: in the case where the target high availability strategy is a master-standby high availability strategy, using the distributed coordination system to monitor the heartbeat data of the multiple master nodes, and when it is determined that the heartbeat data of any master node is abnormal, setting the task status of the master node with the abnormality to be processed, selecting a new master node from the multiple standby nodes, using the new master node to replace the master node with the abnormality, and continuing to execute the target task to be processed corresponding to the master node with the abnormality.

[0010] Optionally, in the case where the target high availability strategy is a multi-active high availability strategy, the use of distributed nodes and the target high availability strategy to preempt the target task whose task status is pending in the task control table includes: dividing the multiple distributed nodes into multiple node groups; obtaining the task identifier of the pending task and the group number of the node group; based on the second calculation relationship between the task identifier and the group number, determining each pending task identifier processed by each node group; for each node group, selecting a master node from the distributed nodes included in the node group, using the main control thread of the master node to construct multiple processing threads for processing tasks, and using the processing thread to preempt the target task corresponding to the task identifier in the task control table.

[0011] Optionally, after selecting the master node from the distributed nodes included in each of the node groups, it further includes: determining one or more standby nodes in each of the node groups; for each of the standby nodes, executing a processing thread for constructing a processing task, and using the processing thread to preempt the target task corresponding to the task identifier in the task control table; wherein the processing progress of the standby node target task is later than the processing progress of the target task of the master node; using the abnormal monitoring strategy corresponding to the target high availability strategy to monitor the task running status of the distributed nodes and perform corresponding task processing operations according to the task running status, including: for the case where the target high availability strategy is a multi-active high availability strategy, using a distributed coordination system to monitor the heartbeat data of the master node in each node group, and when it is determined that the heartbeat data of the master node in any node group is abnormal, determining that the task running status of the master node of the node group is abnormal; determining a new master node from the standby nodes of the node group; and using the new master node to continue to execute the unfinished tasks of the master node, and outputting the result data to the downstream based on the data output breakpoint where the master node abnormality occurs.

[0012] Optionally, the processing of the corresponding data shards by the target task further includes: using distributed nodes to write the processing results and the first processing progress obtained by processing the data of multiple data shards into a message queue, and storing the processing results in a database; when it is determined that there is a storage anomaly in the process of storing the processing results in the database, determining the second processing progress before the processing results are stored according to the data source; according to the second processing progress, determining the unprocessed data after the second processing progress, storing the processing results obtained by processing the unprocessed data in the database, and updating the second processing progress; when the updated second processing progress is consistent with the first processing progress, determining that the data processing satisfies transaction consistency.

[0013] Optionally, the use of distributed nodes to write the processing results obtained by processing the data of multiple data shards into a message queue further includes: after any distributed node determines that the writing of the processing results to the message queue is completed, broadcasting a write task completion message to each downstream node; wherein the task completion message includes the identifier of each distributed node that performs the write operation; after the downstream node determines that the upstream distributed node has completed writing the data to the message queue after receiving the task completion message sent by each distributed node identifier.

[0014] To achieve the above object, according to a second aspect of an embodiment of the present invention, a device for processing distributed data is provided, comprising: a task generation module, a task preemption module and a status determination module; wherein, The task generation module is used to obtain multiple data slices of the data to be processed, construct a corresponding task for each of the data slices, and store the task in the task control table; The preemption task module is used to determine a target high-availability strategy that matches the data to be processed from a plurality of high-availability distributed processing strategies; using distributed nodes and the target high-availability strategy, preempt multiple target tasks whose task status is to be processed in the task control table, and process corresponding data slices through the multiple target tasks; The state determination module is used for the state determination module to monitor the task running status of the distributed nodes and perform corresponding task processing operations according to the task running status by utilizing the abnormal monitoring strategy corresponding to the target high availability strategy.

[0015] Optionally, in the case where the target high availability strategy is a preemptive high availability strategy, the device for processing distributed data is used to utilize distributed nodes and the target high availability strategy to preempt the target task whose task status is pending in the task control table, including: for any distributed node, starting the main control thread of the node, and utilizing the main control thread to generate multiple processing threads for processing tasks according to the number of shards of the data shards; utilizing the main control thread to obtain each of the target tasks to be processed, and storing the target tasks in local storage, and performing a reordering operation on each target task in the local storage, so that multiple processing threads perform the operation of preempting the target task based on the reordered tasks.

[0016] Optionally, in the case where the target high availability strategy is a preemptive high availability strategy, the device for processing distributed data is used to utilize the abnormal monitoring strategy corresponding to the target high availability strategy to monitor the task running status of the distributed nodes and perform corresponding task processing operations according to the task running status, including: registering multiple distributed nodes in a monitoring center; utilizing the monitoring center to perform the following operations based on the abnormal monitoring strategy: monitoring each of the distributed nodes to create different heartbeat files at set time intervals; determining that there is an abnormality in the task running status of the distributed node when it is monitored that the heartbeat file of any distributed node has not been updated at a set number of set time intervals; for the distributed node with abnormal task running status, obtaining the current task processed by the thread of the abnormal distributed node, and changing the task status of the current task to pending in the task control table.

[0017] Optionally, for the case where the target high availability strategy is a master-standby high availability strategy, the device for processing distributed data is used to use distributed nodes and the target high availability strategy to preempt the target task whose task status is pending in the task control table, including: determining multiple master nodes from multiple distributed nodes; obtaining the task identifier of the pending task and the node number of the master nodes; determining each pending task identifier processed by each of the master nodes based on a first calculation relationship between the node number and the task identifier; using the main control thread of each of the master nodes to construct multiple processing threads for processing tasks, and using the processing threads to preempt the target task corresponding to the pending task identifier in the task control table.

[0018] Optionally, a device for processing distributed data is used to determine multiple master nodes from multiple distributed nodes, including: multiple distributed nodes connect to the distributed coordination system at startup, and create a unique directory identifier in a set storage location of the distributed coordination system, and use the distributed node that successfully creates the unique target identifier as the master node; the abnormal monitoring strategy corresponding to the target high availability strategy is used to monitor the task running status of the distributed nodes and perform corresponding task processing operations according to the task running status, including: in the case where the target high availability strategy is a master-standby high availability strategy, the distributed coordination system is used to monitor the heartbeat data of multiple master nodes, and when it is determined that the heartbeat data of any master node is abnormal, the task status of the master node with the abnormality is set to pending, a new master node is selected from multiple standby nodes, and the new master node is used to replace the master node with the abnormality, and the target task to be processed corresponding to the master node with the abnormality continues to be executed.

[0019] Optionally, in the case where the target high availability strategy is a multi-active high availability strategy, the device for processing distributed data is used to use distributed nodes and the target high availability strategy to preempt the target task whose task status is pending in the task control table, including: dividing multiple distributed nodes into multiple node groups; obtaining the task identifier of the pending task and the group number of the node group; based on the second calculation relationship between the task identifier and the group number, determining each pending task identifier processed by each node group; for each node group, selecting a master node from the distributed nodes included in the node group, using the main control thread of the master node to construct multiple processing threads for processing tasks, and using the processing thread to preempt the target task corresponding to the task identifier in the task control table.

[0020] Optionally, the device for processing distributed data, after selecting the master node from the distributed nodes included in each of the node groups, further includes: determining one or more standby nodes in each of the node groups; for each of the standby nodes, executing a processing thread for constructing a processing task, and using the processing thread to preempt the step of the target task corresponding to the task identifier in the task control table; wherein the processing progress of the target task of the standby node is later than the processing progress of the target task of the master node; using the abnormal monitoring strategy corresponding to the target high availability strategy to monitor the task running status of the distributed nodes and perform corresponding task processing operations according to the task running status, including: for the case where the target high availability strategy is a multi-active high availability strategy, using a distributed coordination system to monitor the heartbeat data of the master node in each node group, and when it is determined that the heartbeat data of the master node in any node group is abnormal, determining that the task running status of the master node of the node group is abnormal; determining a new master node from the standby nodes of the node group; and using the new master node to continue to execute the unfinished tasks of the master node, and outputting the result data to the downstream based on the data output breakpoint where the master node abnormality occurs.

[0021] Optionally, the device for processing distributed data is used to process the corresponding data shards through the target task, further comprising: using distributed nodes to write the processing results and the first processing progress obtained by processing the data of multiple data shards into the message queue, and storing the processing results in the database; when it is determined that there is a storage anomaly in the process of storing the processing results in the database, determining the second processing progress before the processing results are stored according to the data source; according to the second processing progress, determining the unprocessed data after the second processing progress, storing the processing results obtained by processing the unprocessed data in the database, and updating the second processing progress; when the updated second processing progress is consistent with the first processing progress, determining that the data processing satisfies transaction consistency.

[0022] Optionally, the device for processing distributed data is used to use distributed nodes to write the processing results obtained by processing the data of multiple data shards into a message queue, and further includes: after any distributed node determines that the writing of the processing results to the message queue is completed, broadcasts a write task completion message to each downstream node; wherein the task completion message includes the identifier of each distributed node that performs the write operation; after the downstream node determines that the upstream distributed node has completed writing the data to the message queue after receiving the task completion message sent by each distributed node identifier.

[0023] To achieve the above-mentioned purpose, according to the third aspect of an embodiment of the present invention, there is provided an electronic device for processing distributed data, characterized in that it comprises: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described in the above-mentioned methods for processing distributed data.

[0024] To achieve the above-mentioned purpose, according to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, any method as described in the above-mentioned method for processing distributed data is implemented.

[0025] To achieve the above-mentioned purpose, according to a fifth aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, characterized in that when the computer program is executed by a processor, any of the above-mentioned methods for processing distributed data is implemented.

[0026] An embodiment of the above invention has the following advantages or beneficial effects: it is able to obtain multiple data slices of data to be processed, construct corresponding tasks for the data slices, and store the tasks in the task control table; determine the target high-availability strategy that matches the data to be processed from a variety of high-availability distributed processing strategies; use distributed nodes and target high-availability strategies to seize multiple target tasks in the task control table whose task status is to be processed, process the corresponding data slices through multiple target tasks, and use the abnormal monitoring strategy corresponding to the target high-availability strategy to monitor the task running status of the distributed nodes and perform task processing operations accordingly. The embodiment of the present invention improves the scenario adaptability, flexibility and scalability of distributed data processing through different high-availability distributed processing strategies; improves the efficiency, reliability, fault tolerance and high availability of processing distributed data.

[0027] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention. Figure 1 It is a flowchart of a method for processing distributed data provided by an embodiment of the present invention; Figure 2A It is a schematic diagram of a flow chart for generating a task control table provided by an embodiment of the present invention; FIG2B is a schematic diagram of a task status change provided by an embodiment of the present invention; Figure 3AIt is a flowchart of processing tasks using a preemptive high availability strategy provided by an embodiment of the present invention; Figure 3B It is a flowchart of processing tasks in a preemptive high availability strategy provided by an embodiment of the present invention; Figure 3C It is another flowchart of handling exceptions in a preemptive high availability strategy provided by an embodiment of the present invention; Figure 3D It is another flowchart of handling exceptions in a preemptive high availability strategy provided by an embodiment of the present invention; Figure 3E It is a flowchart of processing tasks using a master-slave high-availability strategy provided by an embodiment of the present invention; Figure 3F It is a flowchart of a method of handling an exception by using a master-slave high availability strategy provided by an embodiment of the present invention; Figure 3G It is a flowchart of processing tasks using a multi-active high-availability strategy provided by an embodiment of the present invention; Figure 3H It is a flowchart of another method of processing tasks using a multi-active high-availability strategy provided by an embodiment of the present invention; Fig. 3I It is a flowchart of a method of handling an exception by using a multi-active high-availability strategy provided by an embodiment of the present invention; Figure 3J It is a flowchart of a data fault tolerance method provided by an embodiment of the present invention; Figure 3K It is a flowchart of another data fault tolerance method provided by an embodiment of the present invention; Figure 3L It is a flowchart of processing a task completion message provided by an embodiment of the present invention; Figure 4 is a schematic diagram of the structure of a device for processing distributed data provided by an embodiment of the present invention; FIG5 is a diagram of an exemplary system architecture to which an embodiment of the present invention may be applied; Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0030] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information in the financial data involved are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.

[0031] Distributed data processing systems have diverse scenarios, with different data sources, formats, and data stability. Different programs often need to be developed to meet the data processing needs of different scenarios. In order to meet the problem of system fault tolerance in distributed scenarios, high-availability mechanisms often need to be designed when developing distributed data processing programs, and high-availability scenarios for different data scenarios must also be handled separately. Developing multiple corresponding programs increases development costs and makes later maintenance and management more difficult.

[0032] In view of this, if Figure 1 As shown, an embodiment of the present invention provides a method for processing distributed data, which may include the following steps: Step S101: Acquire multiple data slices of the data to be processed, construct a corresponding task for each of the data slices, and store the task in a task control table.

[0033] Specifically, the data to be processed (for example, financial data) often has a large amount of data. Therefore, before processing the data to be processed, an operation of dividing the data to be processed into multiple data shards is performed. Specifically, data sharding information for the data can be configured in a data table of the database (for example, a sharding definition table), or data sharding information for the data shards can be configured through a configuration file; for the scenario of processing data in a database, the data shards can be logical shards of the data table, or physical shards provided by the database; for the scenario of processing data in a message queue, the data shards can be shards defined when the message queue creates a message topic; the present invention improves the flexibility and scalability of distributed data processing by configuring information on data sharding for different types of data sources (databases or message queues), and obtaining multiple data shards of the data to be processed by using the configuration information.

[0034] Furthermore, if Figure 2AAs shown, in the embodiment of the present invention, when the system starts, the initialization program will be called to build a corresponding task for each of the data slices, and the task will be stored in the task control table, wherein the task control table may include data slice information corresponding to the data slice (for example, a slice definition table) and a task identifier of the task for processing the data slice. That is, a corresponding task is built for each of the data slices, and the task is stored in the task control table. It can be understood that the task status of each task in the task control table before processing is "pending".

[0035] like Figure 2B As shown, the life cycle of a task from creation, running to completion mainly includes the following task states: pending, paused, running, completed, and abnormally ended; among them, "pending" means that the task has not started to be processed; "paused" means that the task is prohibited from being preempted, or the preempted task is running, and the processing thread suspends running after receiving a suspend instruction; "running" means that the task has been preempted, and an idle thread is created / reused for processing; if there is no idle thread, it waits for execution in the blocking queue; "completed" means that the task has been preempted and the task has been successfully executed; or the preempted task is running, and the task ends after receiving a forced completion instruction; or the task is not preempted, the task has not started processing, and the task ends after receiving a forced completion instruction; "abnormally ended" means that the task has been preempted and the task has been abnormally executed; or the task has been preempted, the task is running, and the task ends after receiving a cancel execution instruction; or the task is not preempted, the task has not started processing, and the task ends after receiving a cancel execution instruction. Figure 2B The flow relationship between different task states is shown.

[0036] Furthermore, an embodiment of the present invention provides a control instruction table for controlling the task status of a task. After the system is started, the main control thread is used to scan the corresponding control instruction table at a set time interval to obtain the task status control instruction. Among them, the main control thread is used to load the common business static parameter data in the process, manage the task thread, perform heartbeat processing, etc. The life cycle of the main control thread runs through the entire business processing process. The task thread is used to respond to data processing instructions and execute business program logic. The task thread is created by the main control thread and ends after the thread itself is executed. Among them, the process is the smallest unit of resource allocation, and the thread is the actual running unit in the process; a program forms a process after startup. If the process contains multiple threads, the program is a multi-threaded program.

[0037] Furthermore, control instructions are divided into the following categories: task pause, task resume, task forced completion, and task cancellation. After the main control thread obtains the control instruction from the database, it will control the currently executing task thread, and the corresponding task thread will change its state after processing a batch of data currently being processed. Specifically, the corresponding influence relationship of the task state control instruction on the task state is shown in Table 1: The embodiment of the present invention builds a task control table for the task of processing the data slices, thereby using the task control table to manage the task status of the task during the processing, thereby improving the accuracy, flexibility and efficiency of task management.

[0038] Step S102: Determine a target high availability strategy that matches the data to be processed from a variety of high availability distributed processing strategies; utilize distributed nodes and the target high availability strategy to seize multiple target tasks whose task status is to be processed in the task control table, and process corresponding data shards through the multiple target tasks.

[0039] Step S103: using the abnormal monitoring strategy corresponding to the target high availability strategy, monitoring the task running status of the distributed nodes and performing corresponding task processing operations according to the task running status.

[0040] Specifically, in the embodiments of the present invention, a variety of high-availability distributed processing strategies are provided, including: preemptive high-availability strategy, active-standby high-availability strategy, multi-active high-availability strategy, etc. For a system, the target high-availability strategy used can be configured through a configuration file, that is, the target high-availability strategy is any one of the high-availability strategies such as preemptive high-availability strategy, active-standby high-availability strategy, multi-active high-availability strategy, etc. Different high-availability strategies can be configured for different application scenarios and application systems without changing hardware devices such as distributed nodes (that is, different high-availability strategies reuse existing system hardware), which improves the scalability, flexibility and versatility of the system in processing distributed data.

[0041] Furthermore, different high-availability distributed processing strategies have corresponding exception monitoring strategies. The exception monitoring strategy corresponding to the target high-availability strategy is used to monitor the task running status of the distributed nodes and perform corresponding task processing operations according to the task running status. This improves the data reliability and high availability of distributed data processing. This overcomes the problem that the existing method causes the entire system to process abnormally when any computing node has an abnormality, resulting in poor system fault tolerance.

[0042] For the case of the target high availability strategy preemptive high availability strategy, in the preemptive high availability strategy, each process is equivalent, and each process can preempt tasks with a task status of "pending" according to the task control table. In each distributed node, a main control thread is started after the system starts. The main control thread creates threads according to the number of processing threads (see formula 1). Each thread takes the preempted tasks separately to ensure balanced preemption of each business process (each process is evenly assigned tasks).

[0043] Formula 1: Number of processing threads = min (number of data shards / number of all business processes, maximum number of configured threads), where min represents the minimum of the two values.

[0044] Figure 3A FIG. 4 shows a flow chart of a process of processing a task using a preemptive high availability strategy provided by an embodiment of the present invention; Figure 3A As shown, the distributed nodes include node 1 and node 2, and the task status table includes 6 tasks. The task identifiers (task names) of the 6 tasks are 1 to 6 respectively, and the task status of each task is "pending". Assuming that the number of all processes of node 1 and node 2 is 2, and the maximum number of threads is 5 (where the maximum number of threads is configurable), min (6 / 2, 5) is calculated according to formula 1 to get 3, that is, the distributed node identifier is the main control thread of node 1 or node 2 (that is, Figure 3A The main thread in ) creates 3 processing threads ( Figure 3A jobA-Thread-1, jobA-Thread-2, jobA-Thread-3 in .

[0045] Furthermore, in the preemptive high availability strategy, when executing the preemptive task operation, the corresponding task record can be locked through the database lock mechanism. The first one to be preempted will be processed first. In order to avoid all distributed nodes (single instances after the distributed deployment of this system) grabbing the same task and causing performance degradation, the main control program of the main control thread of the distributed node can obtain all pending task data to local storage (such as memory), and then use the shuffle algorithm to rearrange the tasks, and then each processing thread will preempt them (so that multiple processing threads can execute the operation of preempting the target task based on the reordered tasks), such as Figure 3A As shown in the figure, for the task order of "123456", the order obtained by node 1 after reordering (shuffle algorithm) is "125463", and the order obtained by node 2 after reordering (shuffle algorithm) is "534126". The task reordering and preemption operations of each distributed node improve the efficiency of preemption and the uniform distribution of processing data.

[0046] Figure 3BA schematic diagram of a process for processing tasks in a preemptive high availability strategy provided by an embodiment of the present invention is shown; Figure 3B As shown in the figure, the three processing threads preempt three pending tasks (i.e., target tasks) from all tasks (six pending tasks), that is, process the data of the corresponding three data slices. Different tasks are processed after being preempted by multiple processing threads. After each processing thread preempts the corresponding task, it updates the status in the task control table from "pending" to "processing" (or running), and continues to preempt the next task data after processing the current task data. If the processing process fails, other processes can preempt the task, obtain the original program processing progress, and continue calculation and output. That is, in the case where the target high availability strategy is a preemptive high availability strategy, the distributed nodes and the target high availability strategy are used to preempt the target tasks whose task status is pending in the task control table, including: for any distributed node, starting the main control thread of the node, and using the main control thread to generate multiple processing threads for processing tasks according to the number of shards of the data shards; using the main control thread to obtain each of the pending target tasks, and storing the target tasks in local storage; and performing a reordering operation on each target task in the local storage, so that multiple processing threads perform the operation of preempting the target tasks based on the reordered tasks.

[0047] Furthermore, in an embodiment of the present invention, a corresponding fault-tolerant method is provided for the preemptive high-availability strategy. Figure 3C is another flowchart of processing an exception in a preemptive high availability strategy provided by an embodiment of the present invention; specifically, Figure 3C As shown, in an embodiment of the present invention, a monitoring center can be set up, and each distributed node can be monitored by the monitoring center. Each distributed node constructs a heartbeat file in the device to which it belongs according to a set time interval (the time point and content of each constructed heartbeat file are different); further, a separately deployed monitoring center is used to check the heartbeat file. If it is found that the heartbeat file of any distributed node (assuming it is node 2) has not been updated within a set number (for example, 4) of heartbeat intervals (set time intervals), the monitoring center remotely checks the process identifier (PID) of the distributed node to determine whether the process of node 2 is alive. If it is not alive, it is determined that there is an abnormality in node 2, and the monitoring center is used to change the task status being processed by node 2 from "running" to "pending", so that other nodes can preempt the "pending" tasks by scanning the task control table. After other processes scan the tasks to be processed, they will create new task threads for processing. When the number of threads reaches the configured maximum number of threads, it stops. When a task is executed and the thread is released, it will continue to look for pending tasks to preempt; as Figure 3DAnother flowchart of processing an exception in a preemptive high availability strategy is provided in an embodiment of the present invention as shown; Figure 3D As shown, after node 2 fails and exits, node 1 takes over the two tasks 3 and 4 executed by node 2, and creates task threads for tasks 3 and 4 to process the corresponding data shards. Since the number of threads of node 1 reaches the maximum number of threads (for example, 5), the thread preemption is suspended, and task 5 is preempted for processing after the task execution is completed. That is, in the case where the target high availability strategy is a preemptive high availability strategy, the abnormal monitoring strategy corresponding to the target high availability strategy is used to monitor the task running status of the distributed nodes and perform corresponding task processing operations according to the task running status, including: registering multiple distributed nodes in the monitoring center; using the monitoring center to perform the following operations based on the abnormal monitoring strategy: monitoring each of the distributed nodes to create different heartbeat files according to the set time interval; when the heartbeat file of any distributed node is not updated within the set time interval of the set number of monitored distributed nodes, it is determined that the task running status of the distributed node is abnormal; for the distributed node with abnormal task running status, the current task processed by the thread of the abnormal distributed node is obtained, and the task status of the current task is changed to pending in the task control table.

[0048] In an embodiment of the present invention, for the preemptive high availability strategy, when a distributed node fails to write a heartbeat file, but the node is still processing a data task, the monitoring center considers that the node is faulty. Further, in order to prevent two nodes from processing a data shard at the same time, for a certain distributed node, it can be set that after the number of heartbeat write failures reaches the number specified in the configuration file, the node process will actively terminate and exit.

[0049] When the target high availability strategy is a master-slave high availability strategy, the distributed nodes will be divided into master nodes and slave nodes when they are started. The master node processes data, and the slave node is in a ready state. If an exception (failure) occurs in the master node, a new master node can be selected from the slave nodes to replace the abnormal (failed) master node to continue data processing.

[0050] Furthermore, the number of master nodes M corresponding to the master-standby high-availability strategy can be configured in the configuration file, and the number of processes started more than M can be deployed when the system is deployed, wherein the number of started processes is used to compete for the master node from the distributed coordination system (e.g., Zookeeper). When the system is started, all distributed nodes will connect to the distributed coordination system and try to create a sub-path numbered N (1<=N<=M) under a specific path. If the creation is successful, it means that the distributed node becomes the master node No. N. If the creation fails and it is found that the number of sub-paths has not reached M, it will continue to try. After the creation fails and it is found that the number of sub-paths has reached M, the distributed node automatically becomes a standby node, and at the same time, the listener is turned on to listen to all sub-nodes under this path. When it is detected that there is a path deletion, it will try to create the path again to become the master node. That is, the determination of multiple master nodes from the multiple distributed nodes includes: the multiple distributed nodes connect to the distributed coordination system at startup, and create a unique directory identifier in the set storage location of the distributed coordination system, and the distributed node that successfully creates the unique target identifier is used as the master node.

[0051] Figure 3E FIG. 4 shows a flow chart of a task processing method using a master-slave high availability strategy provided by an embodiment of the present invention; Figure 3E As shown, for example, the number of configured master nodes is 2, and the number of processes started is 3. When different processes are started, they connect to the distributed coordination system to execute the operation of competing for the master node (grabbing the master). The successful master nodes are, for example, master node 1 and master node 2, and the failed master node becomes the standby node. Furthermore, in the master-standby high availability strategy, if the master node fails, the standby node needs to replace the master node; further, each master node calculates the target task that it needs to process based on its own node ID, and creates a corresponding number of processing threads to process the data shards corresponding to the task. The master node uses the following calculation formula to calculate the tasks and task identifiers that need to be processed.

[0052] Calculation formula (i.e., the first calculation relationship): N%M=(node ​​ID-1) task N. The task ID is N, and the total number of nodes is M.

[0053] exist Figure 3EIn the schematic diagram shown, the node IDs are "1" and "2". For example, through calculation, node 2 identifies the pending tasks to be processed as "1, 3, 5"; node 1 identifies the pending tasks to be processed as "2, 4, 6"; and the standby node is in a ready state (not processing tasks). The master node further executes the construction of multiple processing threads for processing tasks using the main control thread of each of the master nodes, and uses the processing threads to preempt the target task corresponding to the task identifier in the task control table. That is, for the case where the target high availability strategy is a master-standby high availability strategy, the use of distributed nodes and the target high availability strategy to preempt the target task whose task status is pending in the task control table includes: determining multiple master nodes from multiple distributed nodes; obtaining the task identifier of the task to be processed and the number of master nodes; determining the processing tasks of each of the master nodes and the task identifier corresponding to the processing tasks based on a first calculation relationship between the number of nodes and the task identifier, and determining the processing tasks of each of the master nodes and the task identifier corresponding to the processing tasks; using the main control thread of each of the master nodes to construct multiple processing threads for processing tasks, and using the processing threads to preempt the target task corresponding to the task identifier in the task control table.

[0054] Further, Figure 3F FIG. 1 shows a flow chart of a method for handling an exception by using a master-slave high availability strategy provided by an embodiment of the present invention; Figure 3F As shown, when the distributed coordination system monitors the abnormal heartbeat of any master node (node ​​1), it determines that the master node is abnormal (failed), deletes the path information corresponding to the master node, and notifies all monitored standby nodes, so that each standby node restarts the step of trying to become the master node. After selecting a new master node (i.e., the new node 1), the new master node replaces the failed master node and continues to process the corresponding task. That is, the abnormal monitoring strategy corresponding to the target high availability strategy is used to monitor the task operation status of the distributed node and perform the corresponding task processing operation according to the task operation status, including: using the distributed coordination system to monitor the heartbeat data of multiple master nodes, and when it is determined that the heartbeat data of any master node is abnormal, the task status of the master node with the abnormality is set to pending, a new master node is selected from multiple standby nodes, and the new master node is used to replace the master node with the abnormality, and the target task to be processed corresponding to the master node with the abnormality is continued.

[0055] In an embodiment of the present invention, under the master-standby high availability strategy, in order to prevent multiple master nodes from processing the same data shard, all master nodes will monitor the connection with the distributed coordination system (Zookeeper). When the master node finds that the connection between itself and Zookeeper is interrupted, it will actively terminate and exit its own process.

[0056] In the case where the target high availability strategy is a multi-active high availability strategy, a node group ID (groupID) is configured for each node when it is started. Only one master node is selected for the process with the same node group ID, and the other nodes are all standby nodes. That is, the standby node of each node group can only serve as the standby node of the master node of its own group. The method for becoming a master node in each node group can also be obtained through a distributed coordination system. The specific method is consistent with the method for becoming a master node described in the target high availability strategy, and will not be repeated here. Furthermore, the business node is divided into a master node and a standby node at startup. The master node and the standby node perform data processing at the same time, but the processing result of the standby node is not output to the outside. If the master node fails, a new master node is selected from the standby nodes to replace the failed node to continue data processing.

[0057] Furthermore, for the multi-active high availability strategy, the tasks (data shards) processed by each node group are the same, and the allocation of tasks (data shards) between different node groups is similar to the method of the active-standby high availability strategy. Specifically, the task identifier (task ID) is taken as the remainder of the total number of groups, and the task identifier obtained by the remainder equal to the group ID-1 (that is, the second calculation relationship) is determined to be the task processed by the node group.

[0058] Figure 3G FIG. 4 shows a flow chart of a process for processing tasks using a multi-active high-availability strategy provided by an embodiment of the present invention; Figure 3G As shown, there are two node groups: group 1 and group 2. Group 1 includes a master node and a standby node. Group 2 includes a master node and a standby node. By using the second calculation relationship between the task identifier and the number of groups, for example, determining the pending task identifiers "1, 3, 5" processed by group 2 and the pending task identifiers "2, 4, 6" processed by group 1, Figure 3G In the example, the group IDs are "1" and "2". That is, for the case where the target high availability strategy is a multi-active high availability strategy, the use of distributed nodes and the target high availability strategy to preempt the target task whose task status is pending in the task control table includes: dividing the plurality of distributed nodes into a plurality of node groups; obtaining the task identifier of the pending task and the number of node groups; based on the second calculation relationship between the task identifier and the number of groups, determining the identifiers of the pending tasks processed by each of the node groups; for each node group, selecting a master node from the distributed nodes included in the node group, using the main control thread of the master node to construct multiple processing threads for processing tasks, and using the processing thread to preempt the target task corresponding to the task identifier in the task control table.

[0059] Figure 3HFIG. 4 shows another flow chart of processing tasks using a multi-active high-availability strategy provided by an embodiment of the present invention; Figure 3H As shown, the standby node in a node group performs the same operation on the data slice corresponding to the processing task while the master node processes the data slice corresponding to the task. The difference between the master node and the standby node in the same node group is that the master node is used only to output the data results obtained by processing the data to other units (such as downstream nodes, data sources, etc.), and the standby node does not output the data results obtained by processing the data. Furthermore, the standby node is used to monitor the processing progress of the master node to ensure that its own processing progress will not exceed that of the master node (the processing progress of the target task of the standby node is later than that of the target task of the master node).

[0060] Fig. 3I FIG. 4 shows a flow chart of a method for handling an exception by using a multi-active high-availability strategy provided by an embodiment of the present invention; Fig. 3I As shown, for a certain node group, when Zookeeper determines that an abnormality (failure) occurs in the master node by monitoring the heartbeat data, the standby node obtains the notification of the master node abnormality by monitoring Zookeeper, and then all the standby nodes in the node group execute the operation of striving to become the master node (seizing the master), and the standby node that successfully seizes the master node starts to output data to the downstream from the breakpoint of the master node failure (that is, the new master node is used to continue to execute the unfinished tasks of the master node, and outputs the result data to the downstream based on the data output breakpoint where the master node abnormality occurs).

[0061] That is, after selecting the master node from the distributed nodes included in each of the node groups, it further includes: determining one or more standby nodes in each of the node groups; for each of the standby nodes, executing a processing thread for constructing a processing task, and using the processing thread to preempt the step of the target task corresponding to the task identifier in the task control table, wherein the processing progress of the standby node target task is later than the processing progress of the target task of the master node; using the abnormal monitoring strategy corresponding to the target high availability strategy to monitor the task running status of the distributed nodes and perform corresponding task processing operations according to the task running status, including: using a distributed coordination system to monitor the heartbeat data of the master node in each node group, and when it is determined that the heartbeat data of the master node in any node group is abnormal, determining that the task running status of the master node of the node group is abnormal; determining a new master node from the standby nodes of the node group; and using the new master node to continue to execute the unfinished tasks of the master node, and outputting the result data to the downstream based on the data output breakpoint where the master node abnormality occurs.

[0062] In some data processing scenarios, the calculation results of the data are determined after reading the data shards, and the calculation results are stored in the database and the message queue at the same time. Since the database and the message queue are different storages, there may be data inconsistency problems caused by data storage anomalies. The embodiment of the present invention provides a fault-tolerant processing method for data anomalies in this scenario, which improves the reliability and consistency of data processing.

[0063] Figure 3J A schematic diagram of a data fault tolerance process provided by an embodiment of the present invention is shown; Figure 3J As shown, task 1 reads a batch of data (100 data, represented by 600-699), and then uses the processing thread (jobA-Thread-1) to process the batch of data to obtain the data processing results, and then first stores the data results and data processing progress messages (for example, the last successfully saved processing result is 699, and the first processing progress of task 1 is 699) in the message queue. After successfully storing in the message queue, the data processing results of 600-699 are further stored in the database. If the system exits abnormally when storing in the database, the results and progress saved in the database are all the previous successful progress 599 (i.e., the second processing progress), that is, the data in the message queue and the database are inconsistent, and the data processing progress is inconsistent.

[0064] Furthermore, in the scenario of storing data in a database, data can be read from any location. For example, three processing threads can write 100 data into a data shard at the same time, and the data IDs are 100-199, 200-299, and 300-399 respectively. Since the order of writing to the database is arbitrary, the data of 200-299 may be written first, while the data of 100-199 has not been written yet (i.e., it falls into the database). At this time, if the data is read, only the data of 200-299 can be read. If the reading method is to read in sequence, the data of 300-399 will be read directly after reading 200-299, resulting in the omission of the data of 100-199. For this scenario, in an embodiment of the present invention, a field is added to the business data table: processing task identifier. Through this field, the task ID of the task processing thread will be brought into the database write operation encapsulated by the framework, for example: the task ID corresponding to 100-199 data is A1, the task ID corresponding to 200-299 data is A2, the task ID corresponding to 300-399 data is A3, etc.; that is, the order of distinguishing the written data by the processing task ID is used. Further, when the downstream task processing thread processes the data fragmentation, the data is distinguished by the processing task, and only one upstream processed data is read each time. For example: after the downstream task processing thread obtains the information: A2's data ID is 200-299, A1's data ID is 100-199, and A3's data ID is 300-399, when reading data from the database, the data is read according to the order of distinguishing the written data by the processing task ID; because the output of each processing thread itself must be orderly. By splitting the progress to each task processing thread, this design can ensure that all processing threads are processed only once. The management method for database writing provided by the embodiment of the present invention improves the high availability of data and overcomes the possible problem of data processing omission.

[0065] Figure 3K is a flow chart of another data fault tolerance method provided by an embodiment of the present invention; Figure 3KAs shown, in the case of data inconsistency, when other distributed nodes take over the task (preemptive) or the standby node becomes the master node and continues to process the task (master-standby), the first processing progress can be read from the message queue as 699, and the second processing progress can be read from the database as 599 (that is, the second processing progress before the processing result is stored); further, since there is a sequence relationship between storing the message queue and storing it in the database, breakpoint resumption is started according to the second processing progress (599) of the subsequent database, and then the data from 600 to 699 is read to continue processing. When outputting the data processing result, the result is not output to the message queue, but only saved to the database, and the second processing progress of the database is updated until the second processing progress of the database is consistent with the first processing progress of the message queue to ensure the data. That is, the processing of the corresponding data shards through the target task further includes: using distributed nodes to write the processing results and the first processing progress obtained by processing the data of multiple data shards into the message queue, and storing the processing results in the database; when it is determined that there is a storage abnormality in the process of storing the processing results in the database, determining the second processing progress before the processing results are stored according to the data source; according to the second processing progress, determining the unprocessed data after the second processing progress, storing the processing results obtained by processing the unprocessed data in the database, and updating the second processing progress; when the updated second processing progress is consistent with the first processing progress, determining that the data processing meets the transaction consistency.

[0066] Furthermore, with respect to data processing of streaming data, since the reception and reading of data are dynamic, existing methods for reading batch data cannot determine whether the reading and processing of streaming data is completed. An embodiment of the present invention provides a method for a downstream system to determine whether data writing of an upstream system is completed. Figure 3L FIG. 4 shows a flow chart of a process for processing a task completion message provided by an embodiment of the present invention; Figure 3LAs shown: After task A1 processes data shards (for example, shard 1 to shard 3) and writes the processing results into the message queue, it broadcasts its own task end message to the downstream. The task end message can also include which nodes the job A to which A1 belongs is processed by (for example, processed by two nodes). Downstream tasks B1, B2, and B3 (obtaining the processing end messages for different data shards) receive the end message broadcast by A1, obtain the number of nodes and node identifiers indicated by the end message, and can determine whether the end message of the task of all nodes has been received. The end of job A can be determined. Furthermore, after determining that all upstream end messages have been received, it is determined that if the data source of the downstream output is also a message queue, the current task is further used to broadcast its own end message to the downstream, so that the end message is iteratively transmitted to the downstream in sequence. As a developer, there is no need to develop logic to determine the end of data streaming processing, which improves the efficiency and reliability of data processing. That is, the use of distributed nodes to write the processing results obtained by processing the data of multiple data shards into a message queue further includes: after any distributed node determines that the writing of the processing results into the message queue is completed, broadcasts a write task completion message to each downstream node; wherein the task completion message includes the identifier of each distributed node that performs the write operation; after the downstream node determines that the upstream distributed node has completed writing the data to the message queue after receiving the task completion message sent by each distributed node identifier.

[0067] Specifically, during the task execution process, the task status corresponding to the task in the task control table is updated according to the execution status of the task. Specifically, the description of the task status is consistent with the description of step S101 and will not be repeated here.

[0068] Further, when the task statuses indicated by the task control table are all completed, it is determined that the processing of each data slice of the to-be-processed data is completed.

[0069] Further, the embodiment of the present invention provides a general development interface, which can support batch data processing while supporting streaming data processing. The embodiment of the present invention can be constructed as a java package and published, and the user (developer) of the general development interface of the embodiment of the present invention can implement the following development interfaces by referencing the java package: initialization interface (optional): implement the developer's customized initialization logic; data reading interface: specify to read data from a certain data source (database or message queue) through the development interface; data processing interface: perform developer-defined logic processing on the read data; data output interface: output the processed data to the specified data source (database or message queue); end judgment interface (optional): for batch processing scenarios, this interface does not need to be implemented. In the streaming processing scenario of reading data from the message queue, this interface does not need to be implemented. In the streaming processing scenario of reading data from the database, the developer needs to implement this interface. Processing completion interface (optional): logical operations that need to be performed after all data processing is completed. After implementing these interfaces, it is published as a java program, and the program can be directly run in a distributed manner, thereby realizing the processing flow of task initialization, data reading, processing data, outputting results, judging whether the task is finished, and processing after the task is finished, wherein data processing to judging whether the task is finished is a loop operation. By providing a variety of general development interfaces, developers can directly perform development based on the general development interfaces for specific application scenarios of processing distributed data (such as securities trading data, bank transaction data, etc.), which improves the development efficiency for distributed data processing and reduces development complexity.

[0070] In the embodiment of the present invention, a job represents a Java program implemented using the development interface provided by the embodiment of the present invention. Jobs are usually used to read data for processing and output. In a distributed scenario, a job can be split into multiple tasks for execution, and a task is executed using a task thread in one or more processes as a carrier. The completion of all task processing represents the completion of job processing. When the data processed by the job is split into multiple data slices for parallel execution, the task execution is used to process any data slice.

[0071] In an embodiment of the present invention, an interface based on the http protocol is further provided for an external system to call a REST interface and a database interface (an interface based on a database table for an external system to control the running status of a task through a control instruction table); wherein the functions of the REST interface may include: 1. Job start processing: start the corresponding job according to the Java program name, and obtain the data sharding information. If it is determined that the task control table has not yet been generated for the data sharding information, execute the initialization action to generate the task control table. If the task control table already exists, start creating a task thread and process the corresponding data shard. 2. Job reprocessing: Before starting the job, regardless of whether the task control table is ready, execute the step of clearing the task information of the corresponding job in the task control table and regenerating the task control table, and then start creating a task thread for processing. 3. Check the job status: return the job status of the set job. The relationship between the job status and the task status is shown in Table 2: When the task statuses indicated in the task control table are all successfully completed, it is determined that the processing of each data slice of the data to be processed is completed; the embodiment of the present invention improves the scenario adaptability, flexibility and scalability of distributed data processing, improves the efficiency and reliability of processing distributed data, and provides reliability and data consistency of distributed data processing through fault-tolerant processing solutions for different abnormal scenarios.

[0072] like Figure 4 As shown, an embodiment of the present invention provides a device 400 for processing distributed data, including: a task generation module 401, a task preemption module 402 and a status determination module 403; wherein, The generating task module 401 is used to obtain multiple data slices of the data to be processed, construct a corresponding task for each of the data slices, and store the task in the task control table; The preemption task module 402 is used to determine a target high-availability strategy that matches the data to be processed from a plurality of high-availability distributed processing strategies; using distributed nodes and the target high-availability strategy, preempt multiple target tasks in the task control table whose task status is to be processed, and process corresponding data slices through the multiple target tasks; The state determination module 403 is used to monitor the task running status of the distributed nodes and perform corresponding task processing operations according to the task running status by using the abnormal monitoring strategy corresponding to the target high availability strategy.

[0073] An embodiment of the present invention also provides an electronic device for processing distributed data, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a method for processing distributed data provided in any of the above embodiments.

[0074] An embodiment of the present invention further provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the method for processing distributed data provided by any of the above embodiments is implemented.

[0075] An embodiment of the present invention further provides a computer program product, including a computer program, and when the computer program is executed by a processor, any of the above-mentioned methods for processing distributed data is implemented.

[0076] Figure 5 An exemplary system architecture 500 is shown to which the method for processing distributed data or the apparatus for processing distributed data according to the embodiment of the present invention can be applied.

[0077] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0078] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send requests, etc. Various client applications can be installed on terminal devices 501, 502, 503, such as e-commerce client applications, web browser applications, financial product clients, etc.

[0079] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting various client applications, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0080] The server 505 may be a server that provides various services, such as a background management server that provides support for client applications used by users using terminal devices 501, 502, and 503. The background management server may process the received data request to be processed and feed back the data processing result to the terminal device. Among them, the multiple distributed nodes provided in the embodiment of the present invention may run in the server.

[0081] It should be noted that the method for processing distributed data provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the device for processing distributed data is generally set in the server 505 .

[0082] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0083] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0084] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0085] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0086] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0087] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0088] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0089] The modules and / or units involved in the embodiments of the present invention may be implemented in software or hardware. The modules and / or units described may also be provided in a processor, for example, may be described as: a processor including a task generation module, a task preemption module, and a status determination module. The names of these modules do not, in certain cases, constitute limitations on the modules themselves, for example, the task generation module may also be described as a "module for constructing a corresponding task for each of the data slices".

[0090] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: obtaining multiple data slices of the data to be processed, constructing corresponding tasks for the data slices, and storing the tasks in the task control table; determining a target high-availability strategy that matches the data to be processed from a variety of high-availability distributed processing strategies; using distributed nodes and target high-availability strategies, preempting multiple target tasks in the task control table whose task status is to be processed, processing the corresponding data slices through multiple target tasks, and using the monitoring strategy corresponding to the target high-availability strategy to monitor the task running status of the distributed nodes and perform task processing operations. The embodiments of the present invention improve the scenario adaptability, flexibility and scalability of distributed data processing, and improve the efficiency and high availability of processing distributed data.

[0091] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for processing distributed data, characterized in that: include: Acquire multiple data slices of the data to be processed, construct a corresponding task for each of the data slices, and store the task in a task control table; Determine a target high availability strategy that matches the data to be processed from a plurality of high availability distributed processing strategies; Using distributed nodes and the target high availability strategy, multiple target tasks whose task status is pending in the task control table are seized, and corresponding data slices are processed by the multiple target tasks; The abnormal monitoring strategy corresponding to the target high availability strategy is utilized to monitor the task running status of the distributed nodes and perform corresponding task processing operations according to the task running status.

2. The method according to claim 1, characterized in that In the case where the target high availability strategy is a preemptive high availability strategy, The method of using the distributed nodes and the target high availability strategy to seize the target task whose task status is pending in the task control table includes: For any distributed node, start the main control thread of the node, and use the main control thread to generate multiple processing threads for processing tasks according to the number of shards of the data shards; The main control thread is used to obtain each of the target tasks to be processed, and the target tasks are stored in a local storage, and a reordering operation is performed on each of the target tasks in the local storage, so that multiple processing threads can preempt the target tasks based on the reordered tasks.

3. The method according to claim 1, characterized in that In the case where the target high availability strategy is a preemptive high availability strategy, The using of the abnormal monitoring strategy corresponding to the target high availability strategy to monitor the task running status of the distributed node and performing the corresponding task processing operation according to the task running status includes: Registering a plurality of the distributed nodes in a monitoring center; The monitoring center is used to perform the following operations based on the abnormal monitoring strategy: Monitor each of the distributed nodes to create different heartbeat files at set time intervals; If it is monitored that the heartbeat file of any distributed node is not updated within a set number of set time intervals, it is determined that there is an abnormality in the task operation of the distributed node; For a distributed node with an abnormal task running condition, a current task processed by a thread of the distributed node with the abnormal task is obtained, and a task state of the current task is changed to pending in the task control table.

4. The method according to claim 1, characterized in that: In the case where the target high availability strategy is a master-standby high availability strategy, The method of using the distributed nodes and the target high availability strategy to seize the target task whose task status is pending in the task control table includes: Determine a plurality of master nodes from the plurality of distributed nodes; Get the task ID of the task to be processed and the number of nodes of the master node; Determine, based on a first calculation relationship between the number of nodes and the task identifier, each identifier of the task to be processed by each of the master nodes; A plurality of processing threads for processing tasks are constructed by using the main control thread of each master node, and the processing threads are used to preempt the target task corresponding to the to-be-processed task identifier in the task control table.

5. The method according to claim 4, characterized in that Determining a plurality of master nodes from the plurality of distributed nodes comprises: The plurality of distributed nodes connect to the distributed coordination system at startup, and create a unique directory identifier in a set storage location of the distributed coordination system, and the distributed node that successfully creates the unique target identifier is used as the master node; The using of the abnormal monitoring strategy corresponding to the target high availability strategy to monitor the task running status of the distributed node and performing the corresponding task processing operation according to the task running status includes: In the case where the target high availability strategy is a master-standby high availability strategy, the distributed coordination system is used to monitor the heartbeat data of multiple master nodes. When it is determined that the heartbeat data of any master node is abnormal, the task status processed by the abnormal master node is set to pending, and a new master node is selected from multiple standby nodes. The new master node is used to replace the abnormal master node, and the pending target task corresponding to the abnormal master node continues to be executed.

6. The method according to claim 1, characterized in that In the case where the target high availability strategy is a multi-active high availability strategy, The method of using the distributed nodes and the target high availability strategy to seize the target task whose task status is pending in the task control table includes: Dividing the plurality of distributed nodes into a plurality of node groups; Get the task ID of the task to be processed and the number of node groups; Determine, based on a second calculation relationship between the task identifier and the number of groups, identifiers of each to-be-processed task to be processed by each of the node groups; For each node group, a master node is selected from the distributed nodes included in the node group, a main control thread of the master node is used to construct multiple processing threads for processing tasks, and the processing threads are used to preempt the target task corresponding to the task identifier in the task control table.

7. The method according to claim 6, characterized in that After selecting the master node from the distributed nodes included in each of the node groups, the method further includes: Determine one or more standby nodes in each of the node groups; For each of the standby nodes, executing the step of constructing a processing thread for processing tasks, and using the processing thread to preempt the target task corresponding to the task identifier in the task control table; wherein the processing progress of the target task of the standby node is later than the processing progress of the target task of the master node; The using of the abnormal monitoring strategy corresponding to the target high availability strategy to monitor the task running status of the distributed node and performing the corresponding task processing operation according to the task running status includes: In the case where the target high availability strategy is a multi-active high availability strategy, the heartbeat data of the master nodes in each node group is monitored by using a distributed coordination system, and when it is determined that the heartbeat data of the master node in any node group is abnormal, it is determined that the task running status of the master node of the node group is abnormal; A new master node is determined from the backup nodes of the node group; and the new master node is used to continue to execute the unfinished tasks of the master node, and the result data is output to the downstream based on the data output breakpoint where the master node abnormally occurs.

8. The method according to claim 1, characterized in that: The processing of the corresponding data slices by the target task further includes: Using distributed nodes, writing processing results and the first processing progress obtained by processing the data of the plurality of data slices into a message queue, and storing the processing results in a database; In the case where it is determined that a storage anomaly exists in the process of storing the processing result in the database, determining a second processing progress before the processing result is stored according to the data source; According to the second processing schedule, determining unprocessed data after the second processing schedule, storing processing results obtained by processing the unprocessed data in the database, and updating the second processing schedule; When the updated second processing progress is consistent with the first processing progress, it is determined that the data processing satisfies transaction consistency.

9. The method according to claim 8, characterized in that The step of using the distributed nodes to write the processing results obtained by processing the data of the plurality of data slices into a message queue further comprises: After any distributed node determines that the processing result is written into the message queue, it broadcasts a writing task completion message to each downstream node; wherein the task completion message includes the identifier of each distributed node that performs the writing operation; After the downstream node determines that the task completion message sent by each of the distributed node identifiers is received, the downstream node determines that the upstream distributed node completes writing the data to the message queue.

10. A device for processing distributed data, characterized in that: include: Generate task module, seize task module and determine status module; among them, The task generation module is used to obtain multiple data slices of the data to be processed, construct a corresponding task for each of the data slices, and store the task in the task control table; The preemption task module is used to determine a target high-availability strategy that matches the data to be processed from a plurality of high-availability distributed processing strategies; using distributed nodes and the target high-availability strategy, preempt multiple target tasks whose task status is to be processed in the task control table, and process corresponding data slices through the multiple target tasks; The state determination module is used to monitor the task running status of the distributed node and perform corresponding task processing operations according to the task running status by utilizing the abnormal monitoring strategy corresponding to the target high availability strategy.

11. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images